Vorakl's notes

How to destroy your OS with tar

2024-05-19T20:32:42-07:00

This is a short story about how dangerous a trivial tar unpacking might be, and what can be done to minimize the risk or completely avoid it.

The mistake

Recently, I was practicing an installation of Void Linux via chroot using XBPS method. I needed the XBPS Package Manager installed on my Fedora Linux host to prepare Void Linux's base system. One of the options is to download an archive of statically built tools from the official repository. I chose https://repo-default.voidlinux.org/static/xbps-static-latest.x86_64-musl.tar.xz

$ tar -tf xbps-static-latest.x86_64-musl.tar.xz | head

./
./usr/
./usr/bin/
./usr/bin/xbps-uunshare
./usr/bin/xbps-uhelper
./usr/bin/xbps-uchroot
./usr/bin/xbps-rindex
./usr/bin/xbps-remove
./usr/bin/xbps-reconfigure
./usr/bin/xbps-query

I got so used to having 0:0 as a user:group on all files in archives that I didn't even check their actual permissions and owners. I just looked at the directory structure and noticed that all the executables were conveniently located under the relative path "./usr/bin/". I quickly decided to just extract them to my root directory, so they would be immediately available in my $PATH. This was a big mistake, because if I checked them, I'd see non-standard permissions (700) of a current directory "." and non-standard user:group of the entire archive content:

$ tar -tvf xbps-static-latest.x86_64-musl.tar.xz | head

drwx------ duncaen/netusers  0 2023-09-18 06:37 ./
drwxr-xr-x duncaen/netusers  0 2023-09-18 06:37 ./usr/
drwxr-xr-x duncaen/netusers  0 2023-09-18 06:37 ./usr/bin/
lrwxrwxrwx duncaen/netusers  0 2023-09-18 06:37 ./usr/bin/xbps-uunshare -> xbps-uunshare.static
lrwxrwxrwx duncaen/netusers  0 2023-09-18 06:37 ./usr/bin/xbps-uhelper -> xbps-uhelper.static
lrwxrwxrwx duncaen/netusers  0 2023-09-18 06:37 ./usr/bin/xbps-uchroot -> xbps-uchroot.static
lrwxrwxrwx duncaen/netusers  0 2023-09-18 06:37 ./usr/bin/xbps-rindex -> xbps-rindex.static
lrwxrwxrwx duncaen/netusers  0 2023-09-18 06:37 ./usr/bin/xbps-remove -> xbps-remove.static
lrwxrwxrwx duncaen/netusers  0 2023-09-18 06:37 ./usr/bin/xbps-reconfigure -> xbps-reconfigure.static
lrwxrwxrwx duncaen/netusers  0 2023-09-18 06:37 ./usr/bin/xbps-query -> xbps-query.static

But not knowing that, I ran...

$ sudo tar -C / -xvfp xbps-static-latest.x86_64-musl.tar.xz

In the seconds that followed, I noticed the rapid decline of my system. The windows of my XFCE session stopped redrawing, the X server itself shut down. I couldn't run sudo. I couldn't even boot my system again. It happened so quickly and unexpectedly that I could hardly believe that my last command had caused the crash. Fortunately, booting in a single mode and detailed analysis of the tar archive revealed the root cause.

The root cause

The tar archive contains the current directory "./", which became the root directory when I changed it with "tar -C / ..." to change it before extracting. Restoring the owner and permissions of the current (top) directory of the archive resulted in setting 700 permissions and 2002:2000 as owner:group on my directory tree, which changed its expected state. Thus, my own user completely lost access to the entire file system. Who could have expected that? ;)

For this little demo, I spun up a new VM. Don't try this on your running system!

$ sudo chmod 700 /

$ ls -ld /
drwx------ 17 root root 4096 Mar 27 11:24 /

$ sudo chown 2000:2000 /

$ sudo chown 2000:2000 /usr
-bash: /usr/bin/sudo: Permission denied

$ sudo -s
-bash: /usr/bin/sudo: Permission denied

$ ls -ld /
-bash: /usr/bin/ls: Permission denied

What can be done to prevent it?

In general, it is convenient to create a new archive with a relative directory tree using a command similar to

$ tar -C /path/to/rootfs -czf myarchive.tar.gz .

because you don't have to worry about the internal directory structure, and it's just one command. All files are addressed with simple ".". It is also useful during extraction, since "-C /some/path/" allows you to choose any destination directory. On the other hand, this approach adds a current directory to the archive (the top one in the output above), which takes away all convenience. The default behavior of GNU tar is "Overwrite metadata of existing directories when extracting", which is equivalent to the --overwrite-dir option. For example, if an archive contains a backup of users' home directories with all the necessary permissions, it could be super easy to restore them by running something like "tar -C /home -xpf homes.tar.gz". But this only works if the archive doesn't contain a current directory and the target "/home/" is not modified.

A good way to avoid such pitfalls is to add the --no-overwrite-dir option, which "preserves metadata of existing directories". So, if you run something like "tar -C /home --no-overwrite-dir -xpf homes.tar.gz", all existing directories (including the current one) will remain unchanged!

There are also a few ways to create an archive without a current directory, but most of them require either a directory change beforehand, or defining all files/directories for the future archive. However, I found a way that, although it looks odd, does the job in one command:

$ tar --transform='s|tmp/rootfs|.|' --show-transformed-names -cvf myarchive.tar /tmp/rootfs/*

# or without a verbose mode

$ tar --transform='s|tmp/rootfs|.|' -cf myarchive.tar /tmp/rootfs/*

Thanks to Eric Radman for pointing out that BSD tar has another option, -s, for similar functionality.

Another and pretty typical way to create such archives (packages) is to use fakeroot. It runs as an unprivileged user and pretends that all files are owned by root. In fact, it's just an illusion. Let's have a look at the directory with the extracted original xbps tools:

$ tree -agpu xbps-tools/ | head
[drwxr-xr-x 2002     2000    ]  xbps-tools/
├── [drwxr-xr-x 2002     2000    ]  usr
│   └── [drwxr-xr-x 2002     2000    ]  bin
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-alternatives -> xbps-alternatives.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-alternatives.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-checkvers -> xbps-checkvers.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-checkvers.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-create -> xbps-create.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-create.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-dgraph -> xbps-dgraph.static

And this is how it looks under fakeroot

$ fakeroot /bin/bash

root@localhost> tree -agpu xbps-tools/ | head
[drwxr-xr-x root     root    ]  xbps-tools/
├── [drwxr-xr-x root     root    ]  usr
│   └── [drwxr-xr-x root     root    ]  bin
│       ├── [lrwxrwxrwx root     root    ]  xbps-alternatives -> xbps-alternatives.static
│       ├── [-rwxr-xr-x root     root    ]  xbps-alternatives.static
│       ├── [lrwxrwxrwx root     root    ]  xbps-checkvers -> xbps-checkvers.static
│       ├── [-rwxr-xr-x root     root    ]  xbps-checkvers.static
│       ├── [lrwxrwxrwx root     root    ]  xbps-create -> xbps-create.static
│       ├── [-rwxr-xr-x root     root    ]  xbps-create.static
│       ├── [lrwxrwxrwx root     root    ]  xbps-dgraph -> xbps-dgraph.static

This fake environment allows you to create a tar archive with files owned by root without changing their real owners.

One more nice solution is to use the cpio tool to create or extract POSIX tar archives. This format can be enabled during archive creation by adding "-H ustar". However, during extraction, the format is automatically detected, and it also doesn't change the permissions of the current directory, even if it exists in the archive! If you add the "-d" option and run cpio with sudo, all non-existing subdirectories will be created as root:root, which is also very convenient.

$ tree -agpu newroot/
[drwxr-xr-x root     root    ]  newroot/

$ xz -cd xbps-static-latest.x86_64-musl.tar.xz | sudo cpio -D newroot -idv
.
./usr
./usr/bin
./usr/bin/xbps-uunshare
./usr/bin/xbps-uhelper
./usr/bin/xbps-uchroot
./usr/bin/xbps-rindex
./usr/bin/xbps-remove
./usr/bin/xbps-reconfigure
./usr/bin/xbps-query
./usr/bin/xbps-pkgdb
./usr/bin/xbps-install
./usr/bin/xbps-fetch
./usr/bin/xbps-fbulk
./usr/bin/xbps-digest
./usr/bin/xbps-dgraph
./usr/bin/xbps-create
./usr/bin/xbps-checkvers
./usr/bin/xbps-alternatives
./usr/bin/xbps-alternatives.static
./usr/bin/xbps-checkvers.static
./usr/bin/xbps-create.static
./usr/bin/xbps-dgraph.static
./usr/bin/xbps-digest.static
./usr/bin/xbps-fbulk.static
./usr/bin/xbps-fetch.static
./usr/bin/xbps-install.static
./usr/bin/xbps-pkgdb.static
./usr/bin/xbps-query.static
./usr/bin/xbps-reconfigure.static
./usr/bin/xbps-remove.static
./usr/bin/xbps-rindex.static
./usr/bin/xbps-uchroot.static
./usr/bin/xbps-uhelper.static
./usr/bin/xbps-uunshare.static
./var
./var/db
./var/db/xbps
./var/db/xbps/keys
./var/db/xbps/keys/60:ae:0c:d6:f0:95:17:80:bc:93:46:7a:89:af:a3:2d.plist
./var/db/xbps/keys/3d:b9:c0:50:41:a7:68:4c:2e:2c:a9:a2:5a:04:b7:3f.plist
179893 blocks


$ tree -agpu newroot/ | head
[drwxr-xr-x root     root    ]  newroot/
├── [drwxr-xr-x 2002     2000    ]  usr
│   └── [drwxr-xr-x 2002     2000    ]  bin
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-alternatives -> xbps-alternatives.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-alternatives.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-checkvers -> xbps-checkvers.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-checkvers.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-create -> xbps-create.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-create.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-dgraph -> xbps-dgraph.static

Note that newroot/ was left untouched and is still owned by root:root with 755 permissions. But cpio can do even more. You can create a POSIX tar and easily control which files go in it, because cpio only accepts filenames. So you can get the file list with find and then filter the output to remove (for this particular example) /usr, /usr/bin, /var/, /var/db, and that's it. Super safe and convenient for everyone, while maintaining a relative directory structure inside. Here is an example of how I created a tar archive with cpio, without any "systems" directories, and then extracted it with tar in the usual way:

# Create a tar archive with 'cpio' of previously unpacked xbps tools
$ (cd xbps-tools && find . | grep -v -e '^\.$' -e '^\./usr$' -e '^\./usr/bin$' -e '^\./var$' -e '^\./var/db$' | cpio -ov -H ustar > ../myxbps.tar)
./var/db/xbps/
./var/db/xbps/keys/
./var/db/xbps/keys/3d:b9:c0:50:41:a7:68:4c:2e:2c:a9:a2:5a:04:b7:3f.plist
./var/db/xbps/keys/60:ae:0c:d6:f0:95:17:80:bc:93:46:7a:89:af:a3:2d.plist
./usr/bin/xbps-uunshare.static
./usr/bin/xbps-uhelper.static
./usr/bin/xbps-uchroot.static
./usr/bin/xbps-rindex.static
./usr/bin/xbps-remove.static
./usr/bin/xbps-reconfigure.static
./usr/bin/xbps-query.static
./usr/bin/xbps-pkgdb.static
./usr/bin/xbps-install.static
./usr/bin/xbps-fetch.static
./usr/bin/xbps-fbulk.static
./usr/bin/xbps-digest.static
./usr/bin/xbps-dgraph.static
./usr/bin/xbps-create.static
./usr/bin/xbps-checkvers.static
./usr/bin/xbps-alternatives.static
./usr/bin/xbps-alternatives
./usr/bin/xbps-checkvers
./usr/bin/xbps-create
./usr/bin/xbps-dgraph
./usr/bin/xbps-digest
./usr/bin/xbps-fbulk
./usr/bin/xbps-fetch
./usr/bin/xbps-install
./usr/bin/xbps-pkgdb
./usr/bin/xbps-query
./usr/bin/xbps-reconfigure
./usr/bin/xbps-remove
./usr/bin/xbps-rindex
./usr/bin/xbps-uchroot
./usr/bin/xbps-uhelper
./usr/bin/xbps-uunshare
179889 blocks

$ file myxbps.tar
myxbps.tar: POSIX tar archive

# Check with 'tar' that all files have non root user/group and the archive doesn't contain . /usr /usr/bin /var /var/db
$ tar -tvf myxbps.tar
drwxr-xr-x 2002/2000         0 2024-05-21 16:04 var/db/xbps/
drwxr-xr-x 2002/2000         0 2024-05-21 16:04 var/db/xbps/keys/
-rw-r--r-- 2002/2000      1410 2024-05-21 16:04 var/db/xbps/keys/3d:b9:c0:50:41:a7:68:4c:2e:2c:a9:a2:5a:04:b7:3f.plist
-rw-r--r-- 2002/2000      1410 2024-05-21 16:04 var/db/xbps/keys/60:ae:0c:d6:f0:95:17:80:bc:93:46:7a:89:af:a3:2d.plist
-rwxr-xr-x 2002/2000   5623104 2024-05-21 16:04 usr/bin/xbps-uunshare.static
-rwxr-xr-x 2002/2000   5643584 2024-05-21 16:04 usr/bin/xbps-uhelper.static
-rwxr-xr-x 2002/2000   5631296 2024-05-21 16:04 usr/bin/xbps-uchroot.static
-rwxr-xr-x 2002/2000   6414144 2024-05-21 16:04 usr/bin/xbps-rindex.static
-rwxr-xr-x 2002/2000   5779264 2024-05-21 16:04 usr/bin/xbps-remove.static
-rwxr-xr-x 2002/2000   5643904 2024-05-21 16:04 usr/bin/xbps-reconfigure.static
-rwxr-xr-x 2002/2000   5685440 2024-05-21 16:04 usr/bin/xbps-query.static
-rwxr-xr-x 2002/2000   5643904 2024-05-21 16:04 usr/bin/xbps-pkgdb.static
-rwxr-xr-x 2002/2000   5787648 2024-05-21 16:04 usr/bin/xbps-install.static
-rwxr-xr-x 2002/2000   5639488 2024-05-21 16:04 usr/bin/xbps-fetch.static
-rwxr-xr-x 2002/2000   5631296 2024-05-21 16:04 usr/bin/xbps-fbulk.static
-rwxr-xr-x 2002/2000   5623104 2024-05-21 16:04 usr/bin/xbps-digest.static
-rwxr-xr-x 2002/2000   5640384 2024-05-21 16:04 usr/bin/xbps-dgraph.static
-rwxr-xr-x 2002/2000   6402240 2024-05-21 16:04 usr/bin/xbps-create.static
-rwxr-xr-x 2002/2000   5644032 2024-05-21 16:04 usr/bin/xbps-checkvers.static
-rwxr-xr-x 2002/2000   5643904 2024-05-21 16:04 usr/bin/xbps-alternatives.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-alternatives -> xbps-alternatives.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-checkvers -> xbps-checkvers.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-create -> xbps-create.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-dgraph -> xbps-dgraph.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-digest -> xbps-digest.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-fbulk -> xbps-fbulk.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-fetch -> xbps-fetch.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-install -> xbps-install.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-pkgdb -> xbps-pkgdb.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-query -> xbps-query.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-reconfigure -> xbps-reconfigure.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-remove -> xbps-remove.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-rindex -> xbps-rindex.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-uchroot -> xbps-uchroot.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-uhelper -> xbps-uhelper.static
lrwxrwxrwx 2002/2000         0 2024-05-21 16:04 usr/bin/xbps-uunshare -> xbps-uunshare.static

# Created a new directory to emulate a root file system
$ tree -agpu newroot2/
[drwxr-xr-x root     root    ]  newroot2/
├── [drwxr-xr-x root     root    ]  usr
│   └── [drwxr-xr-x root     root    ]  bin
└── [drwxr-xr-x root     root    ]  var
    └── [drwxr-xr-x root     root    ]  db

# Extract with 'tar' in a usual way
$ sudo tar -C newroot2 -xvf myxbps.tar
var/db/xbps/
var/db/xbps/keys/
var/db/xbps/keys/3d:b9:c0:50:41:a7:68:4c:2e:2c:a9:a2:5a:04:b7:3f.plist
var/db/xbps/keys/60:ae:0c:d6:f0:95:17:80:bc:93:46:7a:89:af:a3:2d.plist
usr/bin/xbps-uunshare.static
usr/bin/xbps-uhelper.static
usr/bin/xbps-uchroot.static
usr/bin/xbps-rindex.static
usr/bin/xbps-remove.static
usr/bin/xbps-reconfigure.static
usr/bin/xbps-query.static
usr/bin/xbps-pkgdb.static
usr/bin/xbps-install.static
usr/bin/xbps-fetch.static
usr/bin/xbps-fbulk.static
usr/bin/xbps-digest.static
usr/bin/xbps-dgraph.static
usr/bin/xbps-create.static
usr/bin/xbps-checkvers.static
usr/bin/xbps-alternatives.static
usr/bin/xbps-alternatives
usr/bin/xbps-checkvers
usr/bin/xbps-create
usr/bin/xbps-dgraph
usr/bin/xbps-digest
usr/bin/xbps-fbulk
usr/bin/xbps-fetch
usr/bin/xbps-install
usr/bin/xbps-pkgdb
usr/bin/xbps-query
usr/bin/xbps-reconfigure
usr/bin/xbps-remove
usr/bin/xbps-rindex
usr/bin/xbps-uchroot
usr/bin/xbps-uhelper
usr/bin/xbps-uunshare

$ tree -agpu newroot2/
[drwxr-xr-x root     root    ]  newroot2/
├── [drwxr-xr-x root     root    ]  usr
│   └── [drwxr-xr-x root     root    ]  bin
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-alternatives -> xbps-alternatives.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-alternatives.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-checkvers -> xbps-checkvers.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-checkvers.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-create -> xbps-create.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-create.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-dgraph -> xbps-dgraph.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-dgraph.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-digest -> xbps-digest.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-digest.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-fbulk -> xbps-fbulk.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-fbulk.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-fetch -> xbps-fetch.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-fetch.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-install -> xbps-install.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-install.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-pkgdb -> xbps-pkgdb.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-pkgdb.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-query -> xbps-query.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-query.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-reconfigure -> xbps-reconfigure.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-reconfigure.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-remove -> xbps-remove.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-remove.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-rindex -> xbps-rindex.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-rindex.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-uchroot -> xbps-uchroot.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-uchroot.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-uhelper -> xbps-uhelper.static
│       ├── [-rwxr-xr-x 2002     2000    ]  xbps-uhelper.static
│       ├── [lrwxrwxrwx 2002     2000    ]  xbps-uunshare -> xbps-uunshare.static
│       └── [-rwxr-xr-x 2002     2000    ]  xbps-uunshare.static
└── [drwxr-xr-x root     root    ]  var
    └── [drwxr-xr-x root     root    ]  db
        └── [drwxr-xr-x 2002     2000    ]  xbps
            └── [drwxr-xr-x 2002     2000    ]  keys
                ├── [-rw-r--r-- 2002     2000    ]  3d:b9:c0:50:41:a7:68:4c:2e:2c:a9:a2:5a:04:b7:3f.plist
                └── [-rw-r--r-- 2002     2000    ]  60:ae:0c:d6:f0:95:17:80:bc:93:46:7a:89:af:a3:2d.plist

Note that all "system" directories such as /usr or /var/db are left unmodified with their original owners and permissions. In fact, you can get the same result with tar either

$ (cd xbps-tools && find . | grep -v -e '^\.$' -e '^\./usr$' -e '^\./usr/bin$' -e '^\./var$' -e '^\./var/db$' | tar --verbatim-files-from -T - -cvf ../myxbps.tar)

That's how I would create such archives with files to be extracted to the root filesystem.

Conclusion

Do not blindly extract an archive if you don't know what it contains! It could be fatal to your system.

A few facts about POSIX

2024-04-23T10:45:58-07:00

TLDR: quick summary of the article

How did we get there?

In the early days of computing, programmers could only dream of portability. All programs were written directly in machine code for each computer architecture they were intended to run on. Assembly languages with mnemonic names for each CPU instruction and other goodies made programmers' lives a little easier, but programs were still architecture-specific. Operating systems (OS) had not yet been invented, so a program not only controlled the entire computer system, it also had to initialize and manage the peripherals. In fact, such bare-metal programs implemented drivers for every device they used. And every time a program needed to run on hardware with a different architecture, it was literally rewritten to accommodate a difference in the CPU instruction set, memory layout, and so on.

This is exactly what happened with Unix, which was originally written in assembly language by Ken Thompson over 50 years ago. The first versions of Unix were written for the PDP-7 platform, and porting it to the PDP-11 meant rewriting the code. When Dennis Ritchie created the C programming language, and together they rewrote most of the Unix code in it, software portability suddenly became possible. There are two main reasons for this. First, the code written in a high-level programming language is platform-agnostic, because compilers translate it into the assembly language for a target architecture. This is even more important for target systems based on RISC CPUs, as they require writing significantly more assembly instructions than CISC CPU architecture. Even porting Unix to another platform was mostly a matter of adapting the architecture-dependent parts of the code. On the other hand, the operating system itself abstracts away all hardware specifics from a user program. Programmers don't have to implement multitasking, memory management, or drivers for different devices as they used to, because it's all part of the OS kernel and runs in the kernel address space. In contrast, user programs run in the user address space and access all of the features provided by the OS through the the system call interface. In Real-time OSes, such as Zephyr OS, it's slightly different, but the idea of memory isolation and protection for user programs is preserved. This leads to two conclusions:

User programs become portable when they are written in a high-level programming language for a particular OS. Once both requirements are met, programs are compiled into instructions for a target CPU and linked with system functions provided by the libc and OS-specific libraries to access the underlying hardware.
Portability is intended to be achieved at the source code level.

The birth of POSIX

This could have been the end of the story, but something fateful happened. Due to a legal restriction, AT&T was not allowed to sell Unix, so there was no money to be made from the newly born OS, which became increasingly popular after it was introduced to the world. However, it turned out to be possible to distribute Unix to any interested organization for the cost of the media. That's how Unix got to Berkeley in 1974 and many other places, leading to the creation of a number of OS derivatives. Some of the best known and still popular today are OSes based on the software distributed by Berkeley (BSD), e.g. FreeBSD and OpenBSD. Despite sharing the same ancestors and principles, each operating system followed its own unique path. Each of these operating systems had a unique interface (API) and implementation of kernel subsystems, syscalls, different system tools, etc. Even libc, which provides common functionality and wrappers on top of syscalls, used to be very OS-specific. All of these OSes were Unix-like, but at the same time, it wasn't possible to take the source code of a program written for one OS and recompile it on another.

Over 35 years ago, these problems with software portability led to the emergence of the first POSIX standard in 1988. The acronym was coined by Richard Stallman, who added "X" to the end of Portable Operating System Interface. The POSIX™ trademark is currently owned by IEEE, and UNIX® is a registered trademark of The Open Group. It's meant to provide a specification of the interface that different Unix operating systems should have in common, including programming languages and tools. It's important to note that the interface is portable, and not the implementation.

This was the common ground that made it possible to compile the same source code of a user program on any OS without modification, if both sides strictly followed the same standard. And this is still true to some extent today, as most modern and widely used Unix-like systems, such as Linux, and *BSD, do not strictly and completely follow POSIX standard, but rather use it as a guide. In addition to POSIX, there is also the Single UNIX Specification (SUS), which was consolidated with a few different POSIX standards in 2001. However, the latest SUS (SUSv4 2018) extends the latest POSIX standard (POSIX.1-2017), which is essentially its base specification, with the X/Open Curses specification. There are a number of operating systems, such as MacOS, which are fully compliant with the POSIX and SUS standards, pass The Open Group conformance tests and can therefore be called Unix operating systems, not just Unix-like. Originally, POSIX was only created for Unix-like OSes, but over time it became so popular that its specification, in the form of the Operating System Abstraction Layer (OSAL), was partially implemented (some subset of the interface that applicable to the target system) in non-Unix OSes, such as Windows, FreeRTOS, Zephyr, etc.

The POSIX spec

The very first standard was ratified by the IEEE in 1988 as IEEE Std 1003.1-1988, so it's called POSIX.1-1988. Since then, the standard has gone through several revisions, with different subsets of the specification being ratified under different names. For example, POSIX.1-1990 (IEEE 1003.1-1990) defined the system interface and computing environment, POSIX.2 (IEEE Std 1003.2-1992) defined command language (shell) and tools, etc. A very good and brief overview of the standard's revisions can be found in the standards(7) Linux man page. You may even come across references to some old revisions, such as POSIX.2, for example, when reading the Bash source code. In 2001, POSIX.1, POSIX.2, and the Single UNIX Specification (SUS) were merged into a single document called POSIX.1-2001. Despite the somewhat misleading name, it does include the shell and tools specifications from POSIX.2. The latest version of the standard is POSIX.1-2017, also known as IEEE Std 1003.1-2017, which is almost identical to POSIX.1-2008.

The document of the standard basically describes a specification that spans over two environments (a build-time and a run-time) and is represented by a few volumes:

Base Definitions: defines common to all volumes general terms and concepts, conformant requirements (symbolic constants, options, option groups), computing environment (locales, regexp, directory structure, tty, environment variables, etc), and C-language header files which need to be implemented by the compliant systems.
System Interfaces: defines the C language standard (ISO C99, ISO/IEC 9899:1999), system service functions, and the extension of the C standard library (libc) in terms of header files and functions.
Shell & Utilities: defines a source code-level interface to the Shell Command Language (sh) and the system utilities (awk, sed, wc, cat, ...), including behavior, command line parameters, exit statuses, etc.
Rationale: includes considerations for portability, subprofiling, option groups, and additional rationale that didn't fit any other volumes.

The current POSIX standard defines source code-level compatibility for only two programming languages: The C language (C99) and the shell command language. However, some of the programs defined under "Utilities", such as awk, also have their own language. Strictly speaking, the C standard library (libc) doesn't have to implement any additional functionality (functions and headers) that is not defined by the C standard (ISO C99 in this case), but most of them do. For example, the ISO C99 standard, defines 24 header files, including math functions (<math.h>), standard input/output (<stdio.h>), date and time (<time.h>), signal management (<signal.h>), string operations (<string.h>), and so on. However, the latest POSIX standard, defines 82 header files and, being fully compliant with ISO C99, extends it with with POSIX threads (<pthreads.h>), semaphores (<semaphore.h>), and many others. Modern libc implementations, e.g. musl libc, are also very OS-specific, providing library functions to access operating system services (wrappers for system calls). Sometimes, the overlap with the POSIX specifications leads to difficulties in implementing the POSIX abstraction layer in the non-Unix operating systems, which also use some portable standalone libc implementations with their own POSIX support, e.g. using picolibc together with Zephyr's POSIX library.

Options and Option Groups

While POSIX standardizes the system interface (C language headers and functions), shell, and utilities, it is not necessary to follow the entire specification to be POSIX conformant. Some features in "POSIX System Interfaces", "POSIX Shell and Utilities", and "XSI System Interfaces" are optional. The <unistd.h> header file contains definitions of the standard symbolic constants for Options, which reflect a particular feature, and Option Groups which define a set of related functions or options. Names of option groups, unlike options, typically do not begin with the underscore symbol. POSIX Conformant systems are intended to implement and support a set of mandatory options with one or more additional options. The symbolic constants for mandatory options should have specific values, e.g. 200809L, while other options may be

undefined or contain -1, which means that the option is not supported for compilation
0, which means the option might or might not be supported at runtime
some other value, which means the option is always supported

These symbolic constants are used by user applications to check the availability of a particular feature. At the C source code-level, constants may be checked either at build time (in #if preprocessing directives) or at runtime, by calling one of the sysconf(), pathconf(), fpathconf(), or confstr(3) functions. In the shell source code, the getconf utility should be used for runtime checks. A very good collection of the POSIX options, their corresponding names for use as the sysconf(3) parameters, and the list of header files and functions that these options represent can be found in the posixoptions(7) Linux man page.

Subprofiling Option Groups are intended for use within the systems where implementing a full POSIX specification is not reasonable. For example, real-time embedded systems are typically resource-constrained, do not have shells, user interfaces, and OS kernels are often designed to run as a single process (with multiple threads). Such systems may only implement subsets of related functions defined by option groups.

Summary

The development of high-level programming languages like C, along with operating systems that abstract away hardware details, enabled software portability at the source code level.
The POSIX standard emerged in 1988 to provide a portable interface specification for Unix-like operating systems, allowing programs to be compiled across different platforms.
The POSIX standard has evolved over time, with the latest version being POSIX.1-2017 (IEEE Std 1003.1-2017).
Modern Unix-like systems like Linux and *BSD do not strictly follow the POSIX standard, but rather use it as a guide.
POSIX standardizes a C API (header files and functions), the shell, and utilities.
POSIX-compliant systems are expected to implement mandatory options and may support additional optional features.
Applications can check for POSIX feature availability at both compile-time and runtime using symbolic constants and system functions.
For resource-constrained systems like real-time embedded platforms, POSIX allows for the implementation of subsets of the full specification through "subprofile" option groups.

How to sort arrays natively in Bash

2024-02-20T18:37:45-08:00

TLDR: quick summary of the article

What would you do if, while implementing some solution in Bash, you suddenly needed to have an array in a sorted order? You might think of the sort tool from the coreutils package. Or you might even think that it's probably a good time to switch to Python or some other language? But it turns out that Bash supports sorting arrays natively! All you need is the asort built-in command. However, it is often not loaded by default, or even packaged on many modern Linux distributions. In this article I'll show you how to build and install Bash with all loadable modules from source, load them, and start writing faster, more advanced Bash scripts with less use of external commands.

First of all, check your Bash version. Version 5.2-release is the target of this article:

echo ${BASH_VERSION}

The built-in loadable modules are loaded with the enable command. Bash expects to find loadable modules in one of the paths specified in the BASH_LOADABLES_PATH environment variable, which is a colon-separated list of directories. Setting this variable and enabling all the necessary commands can be done, for example, with .bashrc. If you are currently running a pre-installed Bash, check that the asort command is not loaded and it cannot be loaded due to its absence:

enable -p | grep asort || { enable -f asort asort && enable -p | grep asort; }

If you see "enable asort" on the screen then the asort builtin is loaded and you can start using it, for example, by checking its help message:

asort --help

Otherwise, let's build it from source. First of all, clone the project's official git repository and enter its directory:

git clone https://git.savannah.gnu.org/git/bash.git && cd bash

The following procedure is pretty standard for any software written in C: you configure the build tools for the specific system, then you build the software, and then you install it on the system. During a configuration step, for example, you can change a default (/usr/local) installation path prefix. I'm going to override it with the same directory as the default. The loadable built-in commands can only be built after the main tool set is built:

./configure --prefix=/usr/local
make
make -C examples/loadables all others
sudo make install
sudo make -C examples/loadables install
sudo cp -v examples/loadables/{necho,hello,cat,pushd,asort} /usr/local/lib/bash/

Loadable built-in commands are installed in /usr/local/lib/bash/ and Bash itself in /usr/local/bin/. The trick with copying files is needed because the asort command is part of the extra commands and, as of this writing and Bash version 5.2.26, the Makefile doesn't support installing it. If all commands finished with no errors, you'll be able to find the loadable commands in the /usr/local/lib/bash/ directory. They are shared objects that can be analyzed in the typical way:

cd /usr/local/lib/bash
ldd asort
file asort

To load built-in commands from these files, you need to know a name of the structure that was defined in the source code. Some files contain only one command, so there is only one such structure, some contain two commands and two structures. You can find out these names by checking the symbol table and looking for the pattern <name>_struct:

$ objdump -t asort | grep _struct
00000000000040c0 g     O .data      0000000000000030              asort_struct

$ objdump -t truefalse | grep _struct
0000000000004020 g     O .data      0000000000000030              false_struct
0000000000004060 g     O .data      0000000000000030              true_struct

Make sure the BASH_LOADABLES_PATH environment variable is set and contains /usr/local/lib/bash, the directory where we installed the built-in commands. Now, everything is ready for testing. Let's run a newly built Bash, and load asort and a few other useful commands, just as an example, using the names we found in the symbol table:

/usr/local/bin/bash
echo ${BASH_VERSION}
echo ${BASH_LOADABLES_PATH}
enable -f asort asort
enable -f truefalse true
enable -f truefalse false
enable -f dsv dsv
dsv --help

Finally, we can perform reverse numerical sorting using only the built-in function which is dsone in-place:

$ declare -a arr=(3 1 15 6 4 5 3)

$ echo ${arr[*]}
3 1 15 6 4 5 3

$ asort -nr arr

$ echo ${arr[*]}
15 6 5 4 3 3 1

Having commands loaded as shared objects allows the Bash to call them directly and avoid creating new processes just to call the external tools with the same functionality. Let's do a quick experiment with mkdir when used as an external tool and loaded into the Bash:

$ strace -e execve /usr/local/bin/bash -c 'mkdir /tmp/mydir'

execve("/usr/local/bin/bash", ["/usr/local/bin/bash", "-c", "mkdir /tmp/mydir"], 0x7ffd7723d6f0 /* 68 vars */) = 0
execve("/usr/bin/mkdir", ["mkdir", "/tmp/mydir"], 0x1e2c010 /* 67 vars */) = 0

$ strace -e execve /usr/local/bin/bash -c 'enable -f mkdir mkdir; mkdir /tmp/mydir2'

execve("/usr/local/bin/bash", ["/usr/local/bin/bash", "-c", "enable -f mkdir mkdir; mkdir /tm"...], 0x7ffd37695000 /* 68 vars */) = 0

You can see that both executables are invoked when mkdir is called as an external tool. But, when mkdir is enabled as a built-in command, there is no an external tool execution, because the Bash calls this function directly. Besides being faster, the asort command has another big advantage over using an external sort tool. Because asort operates on the array data structure directly in memory, you don't have to worry about symbols contained in the array elements and just sort them in-place. They can contain newlines (0x0a or \n) or other bash specific symbols like * or ?:

$ declare -a arr=('**' $'abc\nxyz' $'abc\nefg' '*')

$ declare -p arr
declare -a arr=([0]="**" [1]=$'abc\nxyz' [2]=$'abc\nefg' [3]="*")

$ echo "${arr[1]}"
abc
xyz

$ asort arr

$ declare -p arr
declare -a arr=([0]="*" [1]="**" [2]=$'abc\nefg' [3]=$'abc\nxyz')

It's also worth checking out other loadable commands such as id, ln, mkfifo, cut, cat, stat, tee, uname, and others (see the loadable modules directory). These are fairly common tools used in Bash scripting. They can all be loaded into the Bash itself, resulting in a significant overall performance improvement by eliminating the need to run external commands each time.

Summary

Bash supports sorting arrays natively using the built-in asort command.
The asort and other loadable commands are not enabled by default and may need to be compiled from source.
To build Bash and loadable commands from source, you clone the git repository, configure, make, and install it on your system.
The enable command is used to load builtin commands using their struct names found in the symbol table.
Common loadable commands include asort, truefalse, dsv, id, ln, mkdir, uname, mkdir, and many others.
Loading builtins avoids running external commands, improving performance.
Builtin commands are shared objects that can be analyzed with ldd, file, objdump.
Loadable commands are installed in /usr/local/lib/bash and need BASH_LOADABLES_PATH set to load.

Availability calculation in "nines" notation

2024-02-18T20:50:49-08:00

TLDR: quick summary of the article

The rapidly growing interest in clouds, distributed systems, microservice architecture, and service-oriented applications has led to the emergence of a new branch of computer systems engineering - Site Reliability Engineering (SRE). One of the primary goals of the SRE is to ensure that a service meets certain requirements for production readiness. Services are generally considered to be production when they can be trusted and relied upon. A service provider and the customers, who usually pay for a service, document a common understanding of trust in a Service Level Agreement (SLA). It contains all expectations in the form of Service Level Objectives (SLO) and penalties if these expectations are not met. SLOs are performance and availability goals for a production service, defined on an annual time scale. These are the system characteristics that are both the most valuable to customers and worth committing to keep them within the defined expectations. SLOs are carefully quantified using Service Level Indicators (SLI). SLIs are chosen specifically for SLOs as a measurable form of some properties. It can be a metric or a value derived from logs. SLIs are typically sampled over a much shorter periods of time, from tens of seconds to a few minutes, and then a mean or an average distribution is applied to obtain a value.

Site Reliability Engineers, in turn, are responsible for ensuring that production services meet all target SLOs defined in the SLA. They do this by focusing on the reliability through a set of practices that are more or less standardized across the industry. Some of the most common practices include:

Continuous monitoring of availability and performance characteristics;
Troubleshooting failures and eliminating degradation issues;
Improving overall stability and scalability through automation to keep all key metrics within expected ranges;
Preparing for disaster recovery through continuous stress testing using the error budget, an agreed upon timeframe in which a service can be degraded or unavailable.

Performance SLOs are important goals, but they are only important if a service is available. Availability is so important that it's sometimes mistakenly considered the only SLA component. Finding the right SLI to measure availability can be challenging. It's service-specific and depends on a variety of factors, such as the underlying infrastructure, architecture, etc. In SLO form, availability is expressed as a percentage in what is called "nines" notation. For example, in the clouds, the most common availability SLO is 99.9%, which is called "3-nines". However, you are unlikely to find it higher than 99.999%, or "5-nines". The actual availability of a service in percent is basically calculated as the ratio of the time a service is available to the total uptime (which includes downtime) over the past year.

It is interesting that people who use the nines notation are actually referring to the time when a service is a sort of allowed to be down. This downtime, which is literally allowed by the SLA, forms what is called the error budget. While targeting 100% availability is hardly feasible, it turns out that from a practical point of view, it is more beneficial to commit to a lower availability. Even if all technical possibilities exist to provide more "nines". At certain levels, services with a higher availability will not be noticed by the majority of customers, so it's probably not worth the effort. However, having some error budget opens the doors to experimentation and less stressful deployments of new product features.

It is also useful to know how to estimate a potential downtime, the amount of time when your system may be out of service. To do this, remember that the availability SLO is defined for one-year period. Therefore, 60s by 60m by 24h by 365d gives us 31536000 seconds of a total uptime. Then, if the availability is "five-nines" (99.999%), then the downtime is 0.001%, or 31536000 * 0.001% => 31536000 * 0.00001 = 315.36 sec, which is about 5.256 minutes per year that the service can be down. A similar calculation for "three-nines" (99.9%) availability shows that the service can be down for 31536000 * 0.001 = 31536 seconds, or 525.6 minutes, or 8.76 hours per year.

Summary

Site Reliability Engineering (SRE) focuses on ensuring production services meet requirements for production readiness and can be trusted and relied upon.
A Service Level Agreement (SLA) contains expectations in the form of Service Level Objectives (SLOs) and penalties if not met.
SLOs define annual performance and availability goals for production services.
Service Level Indicators (SLIs) are metrics chosen to measure SLOs, sampled over short periods like seconds to minutes.
SREs ensure services meet SLOs through standardized practices like monitoring, emergency response, and capacity planning.
Availability is the most important SLA component and is expressed as percentages or "nines" denoting hours of annual downtime allowed.
The 99.9% availability SLO allows 8.76 hours of annual downtime while 99.999% allows 5.256 minutes.
Allowing some downtime forms an "error budget" even if 100% uptime is technically possible.
Higher availability beyond a certain level may not be noticeable to most customers.
Calculating allowed downtime involves determining the total seconds in a year and applying the percentage downtime allowed.

A little mess with function parameters in Python

2024-02-17T11:03:29-08:00

TLDR: quick summary of the article

At first glance, Python functions look like those in most other languages, and they behave just as you'd expect. They take arguments, have default values, and can also return a value. This is intentional, of course. But once you dive deeper, you'll see how many specific nuances are hidden internally, providing a programmer with a number of features that make using functions in Python a much more powerful experience. Knowing the differences is critical to understanding why they behave the way they do, so you can get the most out of them.

One of the key features is that functions in Python are objects that are created as soon as they are defined. This allows you to use functions as arguments in other functions or as return values, just like any other Python object. Functions' lifetime is different from the execution time, and they exist even after execution has finished. Functions, being objects, also have a set of predefined attributes that can be extended at any time, and their state is maintained outside of the execution. Parameters become local variables, which are completely different entities from function attributes, which exist only at execution time. Default values in the function definition can also be expressions, but they are evaluated only once. Function arguments are always passed by value, but the values they contain are references. This is why they're sometimes called pass-by-object-references. This also means that parameters, like any other variable in Python, are untyped, and contain a copy of a reference to an object. Changing a parameter (a local variable) generally doesn't change an object (passed as an argument) itself, but only stores a reference to another object. However, there is still a way to change an object that is passed as an argument, if it is a mutable object and the change is made directly to it rather than to a variable. For example, updating elements of a list or a dictionary.

This tutorial will focus only on parameters, their different types, and various ways to define them. Let's start with the most common: a function definition with 4 parameters (a, b, c, d). No types, just names, with a lifetime during function execution, i.e. they are created on the stack as local variables only during function execution. When the function is called, it gets 4 arguments (w, x, y, z), which are also local variables (live on a stack), but in the calling environment, and contain references to some objects. Python takes these references stored in the arguments (w, x, y, z) and copies them into parameters (a, b, c, d) that live as local variables on a stack in the called environment:

def myfunc(a, b, c, d):
    print(a, b, c, d)

def caller():
    w, x, y, z = 10, 20, 30, 40
    myfunc(w, x, y, z)          # 10 20 30 40

caller()

When you call myfunc() this way, references to objects stored in arguments are copied as values to parameters according to their position, e.g. the value of w is copied to a, the value of x is copied to b, and so on. This is why such parameters are also called positional parameters - their position defines the value they get. However, you can assign values to parameters in any order by using keyword arguments, i.e. parameter_name=argument:

def myfunc(a, b, c, d):
    print(a, b, c, d)

def caller():
    w, x, y, z = 10, 20, 30, 40
    myfunc(a=z, b=y, c=x, d=w)  # 40 30 20 10

caller()

Although, all the 4 parameters must be defined each time the function is called. This can be avoided by setting default values for the parameters in the function definition. Keyword pairs must always be defined after positional parameters:

def myfunc(a, b, c, d=2):
    print(a, b, c, d)

def caller():
    w, x, y, z = 10, 20, 30, 40
    myfunc(w, c=x, b=y)         # 10 30 20 2
    myfunc(w, z, y)             # 10 40 30 2

caller()

Default values of parameters are stored in the __defaults__ object attribute. Python allows you to do neat tricks, because this attribute is mutable, and you can assign default values directly to the attribute. This is even possible for the parameters that don't have default values in the function definition and normally need to be set on the function call:

def myfunc(a, b, c, d=2):
    print(a, b, c, d)

print(myfunc.__defaults__)      # (2,)

myfunc.__defaults__ = (100, 200, 300, 400)
print(myfunc.__defaults__)      # (100, 200, 300, 400)

# note that arguments are not passed at all!
myfunc()                        # 100 200 300 400

Default values can also be expressions, but are evaluated only once. For example, if a list is assigned as a default value, its object is created and its reference is assigned each time a default value is used. This may not be the behavior you expect, since a mutated list on a previous function call will still be passed as the default parameter value on the next call:

def myfunc(a, b, c, d=[]):
    d.extend((a, b, c))
    print(a, b, c, d)

myfunc(1, 2, 3)                 # 1 2 3 [1, 2, 3]
myfunc(10, 20, 30)              # 10 20 30 [1, 2, 3, 10, 20, 30]

A possible workaround for having an empty list as the default value is to use None instead. This is a singleton, there is always only one instance. Check a parameter for equivalence to None in the code and assign an empty list during a function execution:

def myfunc(a, b, c, d=None):
    if d is None:
        d = []
    d.extend((a, b, c))
    print(a, b, c, d)

myfunc(1, 2, 3)                 # 1 2 3 [1, 2, 3]
myfunc(10, 20, 30)              # 10 20 30 [10, 20, 30]

Positional and keyword parameters can easily coexist in a relatively free form, with the caveat that keyword parameters are always defined after positional parameters. In general, when calling a function, arguments can be passed in a variety of combinations of positional or keyword types, or omitted with a default value:

def myfunc(a, b, c=1, d=2):
    print(a, b, c, d)

myfunc(3, b=30, c=20)             # 3 30 20 2

However, there are ways to force some parameters to be strictly positional, and others to be keyword only. The first is made possible by another nice feature - a variable number of parameters. Python supports packing and unpacking of arguments during a function call, which can be used to pass an arbitrary number of positional and keyword parameters. It has a special syntax for both cases: positional arguments are packed into tuples if there is a parameter prefixed with an asterisk, e.g. *params, and keyword parameters are packed into dictionaries if there is a parameter prefixed with a double asterisk, e.g. **kwparams. Note that keyword parameters or a **kwparams parameter, if defined, should always follow any positional parameters or a *params, if it's defined:

def myfunc(a, b, *params, c=1, d=2, **kwparams):
    print(a, b, c, d)           # 1 2 20 30
    print(params)               # (3, 4)
    print(kwparams)             # {'e': 50, 'f': 60}

myfunc(1, 2, 3, 4, c=20, d=30, e=50, f=60)

Also, note that the params tuple and the kwparams dictionary are both used without asterisks in the code. It even works the other way around. If you have a tuple or a dictionary with some values, you can easily pass them to a function that takes positional or keyword arguments. Just keep an eye on the number of elements:

def myfunc(a, b, c=3, d=4):
    print(a, b, c, d)

args = (1, 2, 10)
kwargs = {'b': 20, 'c': 30, 'd': 40}

myfunc(*args, 40)               # 1 2 10 40
myfunc(1, **kwargs)             # 1 20 30 40

To define a unified function that can take any number of arguments of any type, it should have a definition that packs all types of parameters, e.g. myfunc(*params, **kwparams). In addition, this syntax strictly separates keyword and positional parameters. If a function has any number of unaggregated keyword parameters after aggregating of positional parameters, then they are considered as keyword-only parameters with default values. The equivalent attribute with default values is called __kwdefaults__:

def myfunc(a, b, *params, c=1, d=2, **kwparams):
    pass

print(myfunc.__defaults__)      # None
print(myfunc.__kwdefaults__)    # {'c': 1, 'd': 2}

This syntax makes it possible to have a simpler function definition in case there is no need in an arbitrary number of parameters. Just put an asterisk between positional and keyword parameters:

def myfunc(a, b, *, c=1, d=2):
    print(a, b, c, d)

# this doesn't work anymore
# myfunc(1, 3, 4, 5)

myfunc(1, 3, d=2, c=1)          # 1 3 1 2

Nevertheless, there is some room for improvisation. Positional arguments can still be passed as keywords:

def myfunc(a, b, *, c=10, d=20):
    print(a, b, c, d)

myfunc(b=3, a=4, d=2, c=1)      # 4 3 1 2
myfunc(a=4, b=3, c=1)           # 4 3 1 20
myfunc(4, b=3, d=2)             # 4 3 10 2
myfunc(4, 3)                    # 4 3 10 20

Fortunately, Python has the syntax to strictly separate positional-only parameters (which cannot be passed as a keyword) from positional parameters (which can either be passed by a value or a keyword). Both can have default values, by the way. Just put a slash between them:

def myfunc(a, /,  b=30, *, c=10, d=20):
    print(a, b, c, d)

# this doesn't work anymore
# myfunc(a=1, b=2, c=4, d=3)

myfunc(4, b=3, d=2, c=1)        # 4 3 1 2
myfunc(4, 3, d=2, c=1)          # 4 3 1 2
myfunc(4, c=1, d=2)             # 4 30 1 2
myfunc(4)                       # 4 30 10 20

print(myfunc.__defaults__)      # (30,)
print(myfunc.__kwdefaults__)    # {'c': 10, 'd': 20}

As a good example, let's take a look at a prototype of the built-in sorted function:

sorted(iterable, /, *, key=None, reverse=False)

This means that the first argument should always be passed as a positional-only argument. You can't pass it as iterable=<something> keyword. However, all subsequent arguments should always be defined as keywords-only. This also means that the order of these arguments, as well as how many of them are passed, is not important.

Another good example is the pop method of the list class:

list.pop(index=-1, /)

index is a positional-only parameter, but if omitted, -1 will be passed by default.

Summary

Functions in Python are objects that are created when defined, allowing them to be used as arguments or return values like any other object.
Parameters become local variables during function execution, while function attributes exist outside of execution.
Arguments are passed by value, but parameters contain a copy of the reference. Changing a parameter doesn't change the original object, but changing a mutable object passed as an argument does.
Parameters can be defined positionally or by keyword. Expressions as the default values are evaluated only once at definition.
The __defaults__ attribute stores default values of positional parameters and is mutable, allowing direct assignment.
An asterisk followed by a name (*var) packs positional arguments into a tuple, while a double asterisk followed by a name (**kwvar) packs keyword arguments into a dictionary.
Keyword arguments always follow positional arguments, with defaults filling in omitted values.
The use of an asterisk and a slash together could be described in the following way: <positional-only parameters> / <positional or keyword parameters> * <keyword-only parameters>.
The __kwdefaults__ attribute stores default values of keyword-only parameters that defined after the asterisk.

Using udp-link to enhance TCP connections stability

2024-01-16T18:44:53-08:00

TLDR: quick summary of the article

I recently discovered udp-link, a very useful project for all those guys like me who spend most of their working time in terminals over SSH connections. The tool implements the UDP transport layer, which acts as a proxy for TCP connections. It's designed to be integrated into the OpenSSH configuration. However, with a little trick, it can also be used as a general-purpose TCP-over-UDP proxy. udp-link greatly improves the stability of connections over unreliable networks that experience packet loss and intermittent connectivity. It also includes an IP roaming, which allows TCP connections to remain alive even if an IP address changes.

udp-link is written in C by Pavel Gulchuk, who has a lot of experience in running unreliable networks. Despite being a young project, the version v0.4 shows pretty stable results. Once configured, you won't think about it anymore. Unless you're surprised every time when SSH connections don't break, survive a laptop's sleep mode and connections to different Wi-Fi networks.

In the current architecture, the client-side tool takes data from the standard input and sends it to the server side via UDP. The same copy of the tool takes that data from the network on a specific UDP port and sends it to a TCP service (local or remote from a server-side perspective). The destination TCP service and a UDP listening port on the server side can be specified on the client at startup. Otherwise, a TCP connection will be established with 127.0.0.1:22, and a port will be randomly chosen from a predefined port range. Note that the server firewall should allow the traffic to this port range on UDP. The TCP service can also reside on a different host if the server side is used as a jumpbox. I consider it one of the greatest features that udp-link uses a zero server-side configuration, all configuration tweaks happen only on the client side.

udp-link on the server side does not run as a daemon or listen on a UDP port all the time. Instead, the client initiates the invocation of the tool on the server side in listening mode with a randomly generated key. This key is used to authenticate the client connection. This is done on demand by establishing a normal SSH connection over TCP with the server side, temporarily, just to run the tool in the background. The connection is then closed. This is where a secure client authentication comes into play. udp-link doesn't encrypt the transferred data, which is useful when is used together with SSH because it avoids a double encryption, but needs to be kept that in mind when used with other configurations.

To start using udp-link, you need to clone the repository, compile, and install the tool on both sides

git clone https://github.com/pgul/udp-link.git
cd  udp-link
make
sudo make install

and then make an SSH connection on the client side by executing a command similar to

ssh -o ProxyCommand="udp-link %r@%h" user@host

ProxyCommand allows SSH to send all its data to the standard input of a specified command instead of to a TCP connection. This command will be responsible for sending the data to a server side in some way and should eventually deliver it to a target SSH service. OpenSSH also supports a number of macros such as %r and %p which can be found in its documentation. Personally, I use SSH in a slightly different way and never send out my public SSH keys to unknown hosts. More details on this topic can be found in a great article 'OpenSSH client side key management for better privacy and security', written by Timothée Ravier. So I'm actively using ssh_config files, where I specify all connection-specific details, such as hostname, username, SSH key, and in this case, ProxyCommand. My typical ssh_config file looks something like this

Host some-server
    user some-user
    hostname some-IP
    IdentityFile ~/.ssh/ssh-some-server.key
    ProxyCommand udp-link some-IP

Host some-IP
    user some-user
    IdentityFile ~/.ssh/ssh-some-server.key

Host *
    IdentitiesOnly yes
    IdentityFile /dev/null
    GSSAPIAuthentication no
    HostbasedAuthentication no
    SendEnv no

and then to connect I just run

ssh some-server

The second Host some-IP block is needed to provide a correct SSH key to a temporary SSH connection (without ProxyCommand) that udp-link establishes at the beginning of a new session. To debug the connection, add --debug option

ssh -o ProxyCommand="udp-link --debug some-IP" some-server

If I need to bind a connection to a specific UDP port on the server side, I initiate a connection like this

ssh -o ProxyCommand="udp-link -b 1234 some-IP" some-server

You can also bind it to a privileged port (1-1024), but udp-link needs root permissions to do this, which can be achieved in a number of ways, such as making it root-owned with the setuid bit turned on the server-side copy of a binary file.

chown root /usr/local/bin/udp-link
chmod u+s /usr/local/bin/udp-link

Unlike other projects with a similar goal, e.g. Mosh, udp-link doesn't allocate a pseudo-terminal, which I consider a feature, because it opens the possibility to use the tool not only for accessing remote terminals, but also for proxying any arbitrary TCP connection. However, udp-link cannot currently listen on a local TCP port on the client side. Fortunately, this can be worked around by adding socat and its exceptional ability to connect things. However, socat cannot be paired with udp-link via an unnamed pipe, because pipes provide a unidirectional interprocess communication, while here we need a bidirectional communication to get data back from the network. The trick is that udp-link is invoked by socat. Here is an example of how to open a listening 2525/TCP port on the client side, then proxy a future TCP connection over a UDP channel to a remote host, and connect it to a 25/TCP port on the server's localhost in debug mode

socat TCP-LISTEN:2525 SYSTEM:"udp-link -t 127.0.0.1\:25 --debug some-IP"

udp-link is a small, flexible and very useful tool. I hope to see further development, adding new features and maturing the code base.

Summary

udp-link is a tool that implements the UDP transport layer to act as a proxy for TCP connections over unreliable networks.
It is designed to be integrated into OpenSSH configuration to improve stability of SSH connections.
udp-link allows TCP connections to remain alive even if the IP address changes through its IP roaming feature.
On the client side, udp-link takes data from standard input and sends it to the server side via UDP, where it is then sent to the target TCP service.
The server side of udp-link does not run as a daemon and instead is invoked on demand by the client through a temporary SSH connection.
Authentication is done through a randomly generated key during the temporary SSH connection.
udp-link doesn't encrypt the transferred data, which is useful when is used together with SSH to avoid double encryption.
Installation is done by cloning the GitHub repo, compiling, and installing it on both client and server.
All configuration is done only on the client side

The zoo of binary-to-text encoding schemes

2020-05-13T20:01:53-07:00

In the previous article, I discussed the use of the positional numeral system for the purpose of binary-to-text translation. That method represents a binary file as a single big number with the radix 256 and then converts this big number to another one with an arbitrary radix (base) in a range from 2 to 94. Although this approach gives the minimum possible size overhead, unfortunately, it also has a number of downsides which make it hardly usable in a real-world situation. In this article, I'll show what is used in practice, which encodings could be found in the wild, and how to build your own encoder.

What's wrong with the positional single number encoding?

The main issue with converting a file as a big number in radix 256 to another big number with a smaller radix is that you need to read the whole file, load it to the memory and build actually that big number from each byte of the file. To construct a number, the Least Significant Byte (LSB), which is the last byte of a file, needs to be read and loaded. Although, there is not always enough memory to load a whole file as well as there is not always the whole file is available at any given time. For instance, if it's being transmitted over a network and only a small amount of bytes from the beginning (from the Most Significant Byte, MSB) has been loaded. This issue is usually addressed by processing a file as a stream of bytes, in chunks, which then are being converted in the same way (by converting a number from one base to another). These chunks are much smaller and, ideally, fit the CPU registers' size (up to 8 bytes). The only question here is how to find the best size and ratio of such chunks (input and output) to keep the size overhead as closely as possible to a minimum available by treating files as big numbers.

What's the essence of a positional numeral system?

In the positional numeral systems, everything turns around a radix (base) which shows how many different symbols are used to represent values. The actual glyph doesn't matter. Only their quantity. All these symbols are grouped in an alphabet (a table) where every symbol is defined by its own position, and this position represents its value. As long as counting starts from 0, the maximum symbol's value, in any numeral system, is always radix - 1. For instance, in the numeral system with a radix 10 (Decimal), the maximum value has a symbol '9'. But, for a system with a radix 2 (Binary), the maximum value has a symbol '1'. When symbols from an alphabet appear as a part of a number, they are called digits. A digit's position, in this case, is called index and defines the power of a radix while its value (position in the alphabet) defines a coefficient within the power of that radix.

The first crucial conclusion here is that any number, represented in some positional numeral system, gets its meaning only when is known its radix.

The second conclusion is not so obvious. Humans in most cases nowadays use the Decimal numeral system. Numbers gain more sense for them when they are represented as Decimal numbers and this is the system that is used the most for calculations. To any symbol in an alphabet is assigned its certain position which is a number with some radix. In most cases, this radix is 10 (Decimal). The Decimal numeral system is a temporary system that is used for converting one numeral system to another. Every time, when a number is defined by a radix, this radix is Decimal, no matter what's the radix of a number. Every time, when there is a need to convert a number X with radix M to a number Y with radix M, both numbers (X and Y) are represented by some certain alphabets (which define symbols with values), but their radixes (M and N) are always represented in Decimal system, thus, Decimal system is used as an intermediate numeral system to which a number X is converted first, and then the intermediate number is converted to a number Y. The intermediate numeral system could have been any radix, but radix 10 is what people use for calculations and that's what can be found in most converters implementations.

The third conclusion is even more important. Symbols don't bring any value, only their position in the alphabet. This means we need to know not only an actual number's representation but also its radix and an alphabet - the table that contains symbols assigned to values (position within the table). A good example is an alphabet of 16 symbols for Hexadecimal numbers (radix 16). There are first 10 digits linked to equivalent values, so the symbol '0' is linked to 0, '1' to 1, and so on up to the symbol '9' linked to 9. The rest 6 values (from 10 to 15) linked to English letter symbols (from 'A' to 'F'). And again, these values (positions in the table) are all Decimal numbers (radix 10). By the way, the table could have been different, but that's what is used by convention, so anyone is able to interpret Hexadecimal numbers in the same way.

Where does the overhead come from?

Let's take a look at a few examples. This is a number '123' that is represented by three symbols, but until we know a radix, it is not possible to understand its value. If the radix is 10 then it is 'one hundred twenty three' in the Decimal system and it can be calculated by the formula for converting a numeral system with any radix to radix 10 (because all numbers in this formula have radix 10): 1*10^2 + 2*10^1 + 3*10^0 = 123. If the radix is 8, then it is an Octal system and it is constructed as 1*8^2 + 2*8^1 + 3*8^0 which gives us a Decimal number 83. So, '123 base 8' equals to '83 base 10'. It is worth noticing that converting a number to a higher radix leads to lower a number of symbols needed for its representation. The converse is also true. If a number 83 with a radix 10 is converted to a radix 2, it gets a form '1010011'. Notice, the radix is changed from 10 to 2 and the number of symbols changed from 2 to 7! As lower a radix gets, as more symbols appear in representation.

Let's get back to binary files. What we can determine as 'symbol representation' or 'digits', 'alphabet', and 'radix' based on a structure of an ordinary file? Any file consists of bytes as it is the minimum addressable group of bits. It cannot be less than 8 bits. So, we can think about a number representation as of some amount of bytes. The chunks can vary from 1 byte to a file's size. For example, if there is only one byte, then the number consists of only one digit. One byte or 8 bits (binary digits with a radix 2) allows one to represent 2^8 = 256 different numbers. That means, we can persist 256 different symbols with their positions to build an alphabet. The good news, such a table has already been standardized many years ago and called ASCII. And the last thing, as the alphabet size is 256 symbols then a radix is also 256. Here is our number: a number of bytes in the chunk that we are going to process are the number of digits, a radix is 256, and the coefficient has a range from 0 to 255. For example, if a group of bytes to read from a stream and process at once consists of 4 bytes (from MSB to LSB): [13, 200, 3, 65] then our number can be represented as a Decimal number (radix 10) as 13*256^3 + 200*256^2 + 3*256^1 + 65*256^0 = 231211841

As it was discussed in the previous article, we can use no more than 94 different symbols to reliably represent texts. Thus, the desired radix lies somewhere in the range from 2 to 94. Even 94 is much less than 256, so a number's representation in a new radix is likely to have more symbols. This means, in turn, that the output group will have more bytes as it is a minimum amount of data we can operate on, even if a digit represented by a symbol needs fewer bits. You'll still need to allocate the whole byte for each symbol in the new radix number representation. Some amount of bits in such bytes will never be used. This is the root of inefficiency, and that's why it's highly important to find a good ratio of output to input byte groups. For instance, the most used nowadays Base64 encoding converts binary files to texts by reading 3-bytes groups from the input stream, represents them as a 3-digits number with a radix 256 (log[256^3, 2] = 24 bit), and then converts this number to a 4-digits number with a radix 64 (log[64^4, 2] = 24 bit), which in turn is written to the output stream as a group of 4 bytes. So, the ratio of output to input is 4/3 = 1.333333. In other words, the size overhead is 33.(3)%. There are a few considerations behind the logic of choosing the exact combination of input and output groups for a streaming conversion, which includes a target radix, a desirable/available alphabet, an ability to natively compute on a CPU, etc.

How to calculate a minimal overhead?

Let's calculate first, how many digits of a target base (radix) are needed to represent exactly the same number in the initial base. For instance, there is given a number 123 with a radix 10. How many bits (binary digits, a radix 2) are needed to represent the same decimal number? Every digit is a coefficient of power of a base. If it is not enough, one more base is added in power +1 to finally construct a number. Keeping in mind that counting starts from 0, if it's said that to represent some number 8 bit are needed, this means all bases in powers from 0 to 7 with their coefficients have to be summed up. Thus, to find out a number of digits needed to represent the number in some radix, we need to find an exponent, to which a new radix needs to be exponentiated. In our case, for a base-10 number 123, we need to calculate an exponent of a base-2 by using a logarithm function: log[123, 2] = 6.9425145. This means, to represent a number 123 with base 10, a little bit less than 7 bits will be enough. All computer systems operate on a set of natural numbers only. It is not possible to use 6.9425145 bits as this number is an approximated value of needed bits. 6 bits apparently won't be enough (2^6 = 64, which is much less than 123), so the only right approach is always to round up (by calling a ceil function) any non-integer values. Unfortunately, 7 bits are able to represent a bigger number (2^7 = 128) and this again contributes to a final overhead.

Let's have a look at the Base64 again. We know already (but not why is that, yet), that this streaming system uses 3 input bytes (a 3-digit number with a base 256) and converts them to a number with a base 64. How many base-64 digits will this number contain? The answer is log[256^3, 64] = 4, four digits, hence 4 symbols from the base64 alphabet.

While looking for the good input and output group sizes it's good to know a theoretically possible minimum of the overhead. To find it out, we need to do a similar calculation but take the minimally possible amount of input data, which is one byte (8 bits, decimal 2^8 = 256). For the Base64, it is log[256, 64] = 1.33(3), that is again 33.(3)%. For the Base32 it is log[256, 32] = 1.6, that is 60%. And for the Base16 it is log[256, 16] = 2, that is 100%. Wow! These theoretical numbers are exactly the same as practically used ratios of output bytes to input bytes give. Here are they: for the Base64 it is 4 / 3 = 1.33(3), for the Base32 it is 8 / 5 = 1.6, and for the Base16 it is 2 / 1 = 2. There is one interesting fact, all these three bases (16, 32, 64) have one thing in common - they all are powers of two! This leads us to the conclusion that converting numbers within the "power of two" bases allows one to get the best possible ratio and match precisely an input bits group to an output bits group. Although it is not always desirable or even possible. Sometimes there is a need to use a specific alphabet, e.g. in Base36, or the minimal overhead, e.g. in Base85 or Base94. All these bases are not the "powers of two", so a tradeoff has to be found to minimize the overhead.

How to calculate optimal input and output groups?

Alright, we've calculated a number of digits needed to represent some number in another base. But, why is that only a theoretical minimum? Why in practice it would need more? And, why would we still need to find a good ratio of output to input byte groups? To answer these questions, let's have a look at the Base85 encoding. To represent 1 byte (Base256) of information in Base85, it needs log[256, 85] = 1.24816852 digits. But, we can't use 1.248 digits. Only positive whole numbers are available! 1 digit is neither possible (too little). Then, 2 digits are the only way to go. In other words, to represent 1 byte (with a number in Base256), in fact, we'd need 2 bytes (with a number in Base85), where ~75% of space will be wasted, as the ratio is 2/1 = 2 and this is a 100% overhead, instead of a theoretical 24.8%. There is no point to use 1-byte input group and 2-bytes output group. Thus, there should be some good input and output groups so their ratio goes as close as possible to a calculated minimum or even match it!

The following approach starts from 1-byte group and using the same formula, every time checks a number of digits in the destination base. if it's not close enough, increments the input group by 1 byte and checks again. You can decide on your own, what is the applicable size of an input group and how close to the whole number up (ceil function) the output group needs to be.

This code goes through all bases, from 2 to 94, and prints a first found input/output group that has a delta between the number of digits and its rounded value less or equal 0.1, if any. That is, ceil(x) - x <=0.1. I limited an input group by 20 bytes but in reality, groups larger than 8 bytes (64bit) will require either a more complicated implementation still based on 64bit variable types or the big number mathematics which would bring it back to the solution from the previous article.

from math import log, ceil

def find_dec_fractions(num):
    for i, k in [(i, log(256**i, num)) for i in range(1,20)]:
        if (ceil(k)-k)<=0.1:
            return (i, k)

for i in range(2, 95):
    try:
        b_in, b_out = find_dec_fractions(i)
    except TypeError:
        continue
    print(f'Base{i}: output/input {b_out} / {b_in}; Ratio: {ceil(b_out)} / {b_in} = {ceil(b_out)/b_in}')

Base2: output/input 8.0 / 1; Ratio: 8 / 1 = 8.0
Base3: output/input 95.90132254286152 / 19; Ratio: 96 / 19 = 5.052631578947368
Base4: output/input 4.0 / 1; Ratio: 4 / 1 = 4.0
Base6: output/input 30.948224578763327 / 10; Ratio: 31 / 10 = 3.1
Base7: output/input 19.94760247804924 / 7; Ratio: 20 / 7 = 2.857142857142857
Base8: output/input 8.0 / 3; Ratio: 8 / 3 = 2.6666666666666665
Base9: output/input 42.9032232428591 / 17; Ratio: 43 / 17 = 2.5294117647058822
Base10: output/input 40.940079410301436 / 17; Ratio: 41 / 17 = 2.411764705882353
Base11: output/input 6.937555831629307 / 3; Ratio: 7 / 3 = 2.3333333333333335
Base12: output/input 8.926174260836154 / 4; Ratio: 9 / 4 = 2.25
Base13: output/input 12.971431412511347 / 6; Ratio: 13 / 6 = 2.1666666666666665
Base14: output/input 18.910766522677935 / 9; Ratio: 19 / 9 = 2.111111111111111
Base15: output/input 38.905619771091956 / 19; Ratio: 39 / 19 = 2.0526315789473686
Base16: output/input 2.0 / 1; Ratio: 2 / 1 = 2.0
Base17: output/input 1.957204336945808 / 1; Ratio: 2 / 1 = 2.0
Base18: output/input 1.9184997325450517 / 1; Ratio: 2 / 1 = 2.0
Base19: output/input 16.949441762397953 / 9; Ratio: 17 / 9 = 1.8888888888888888
Base20: output/input 12.957179936946513 / 7; Ratio: 13 / 7 = 1.8571428571428572
Base21: output/input 10.928171937453742 / 6; Ratio: 11 / 6 = 1.8333333333333333
Base22: output/input 8.969752968703016 / 5; Ratio: 9 / 5 = 1.8
Base23: output/input 15.916660520940269 / 9; Ratio: 16 / 9 = 1.7777777777777777
Base24: output/input 6.97933734353701 / 4; Ratio: 7 / 4 = 1.75
Base25: output/input 18.949768555229294 / 11; Ratio: 19 / 11 = 1.7272727272727273
Base26: output/input 11.913778998988336 / 7; Ratio: 12 / 7 = 1.7142857142857142
Base27: output/input 26.919669485715517 / 16; Ratio: 27 / 16 = 1.6875
Base28: output/input 4.992350344236227 / 3; Ratio: 5 / 3 = 1.6666666666666667
Base29: output/input 4.940323979050427 / 3; Ratio: 5 / 3 = 1.6666666666666667
Base30: output/input 17.933964143964545 / 11; Ratio: 18 / 11 = 1.6363636363636365
Base31: output/input 12.91834154125439 / 8; Ratio: 13 / 8 = 1.625
Base32: output/input 8.0 / 5; Ratio: 8 / 5 = 1.6
Base33: output/input 7.929594526822421 / 5; Ratio: 8 / 5 = 1.6
Base35: output/input 10.917705226052034 / 7; Ratio: 11 / 7 = 1.5714285714285714
Base36: output/input 13.926701060443497 / 9; Ratio: 14 / 9 = 1.5555555555555556
Base37: output/input 19.963706880682256 / 13; Ratio: 20 / 13 = 1.5384615384615385
Base38: output/input 25.91499209004118 / 17; Ratio: 26 / 17 = 1.5294117647058822
Base41: output/input 2.9864385798230937 / 2; Ratio: 3 / 2 = 1.5
Base42: output/input 2.9671843746459023 / 2; Ratio: 3 / 2 = 1.5
Base43: output/input 2.9486213303792987 / 2; Ratio: 3 / 2 = 1.5
Base44: output/input 2.930708014618138 / 2; Ratio: 3 / 2 = 1.5
Base45: output/input 2.913406407519012 / 2; Ratio: 3 / 2 = 1.5
Base46: output/input 15.93174851664354 / 11; Ratio: 16 / 11 = 1.4545454545454546
Base47: output/input 12.96225551928187 / 9; Ratio: 13 / 9 = 1.4444444444444444
Base48: output/input 22.918685664133292 / 16; Ratio: 23 / 16 = 1.4375
Base49: output/input 9.97380123902462 / 7; Ratio: 10 / 7 = 1.4285714285714286
Base50: output/input 9.922293927591243 / 7; Ratio: 10 / 7 = 1.4285714285714286
Base51: output/input 16.92397770133268 / 12; Ratio: 17 / 12 = 1.4166666666666667
Base53: output/input 6.983337201921797 / 5; Ratio: 7 / 5 = 1.4
Base54: output/input 6.9506137148575995 / 5; Ratio: 7 / 5 = 1.4
Base55: output/input 6.918787617803083 / 5; Ratio: 7 / 5 = 1.4
Base56: output/input 17.9083251145862 / 13; Ratio: 18 / 13 = 1.3846153846153846
Base57: output/input 10.97226243673046 / 8; Ratio: 11 / 8 = 1.375
Base58: output/input 10.925265898478088 / 8; Ratio: 11 / 8 = 1.375
Base59: output/input 14.959262233248435 / 11; Ratio: 15 / 11 = 1.3636363636363635
Base60: output/input 18.960906451063522 / 14; Ratio: 19 / 14 = 1.3571428571428572
Base61: output/input 22.93138142177215 / 17; Ratio: 23 / 17 = 1.3529411764705883
Base64: output/input 4.0 / 3; Ratio: 4 / 3 = 1.3333333333333333
Base65: output/input 3.9851435091825076 / 3; Ratio: 4 / 3 = 1.3333333333333333
Base66: output/input 3.9706212940573997 / 3; Ratio: 4 / 3 = 1.3333333333333333
Base67: output/input 3.9564205613318486 / 3; Ratio: 4 / 3 = 1.3333333333333333
Base68: output/input 3.942529199089205 / 3; Ratio: 4 / 3 = 1.3333333333333333
Base69: output/input 3.9289357306851747 / 3; Ratio: 4 / 3 = 1.3333333333333333
Base70: output/input 3.9156292724042583 / 3; Ratio: 4 / 3 = 1.3333333333333333
Base71: output/input 3.9025994945192193 / 3; Ratio: 4 / 3 = 1.3333333333333333
Base72: output/input 12.966121951449782 / 10; Ratio: 13 / 10 = 1.3
Base73: output/input 12.92443739543971 / 10; Ratio: 13 / 10 = 1.3
Base74: output/input 21.90208895887644 / 17; Ratio: 22 / 17 = 1.2941176470588236
Base75: output/input 8.990468784305198 / 7; Ratio: 9 / 7 = 1.2857142857142858
Base76: output/input 8.962972102269996 / 7; Ratio: 9 / 7 = 1.2857142857142858
Base77: output/input 8.935999277516537 / 7; Ratio: 9 / 7 = 1.2857142857142858
Base78: output/input 8.909533240680473 / 7; Ratio: 9 / 7 = 1.2857142857142858
Base79: output/input 13.959876384572452 / 11; Ratio: 14 / 11 = 1.2727272727272727
Base80: output/input 13.919804002700841 / 11; Ratio: 14 / 11 = 1.2727272727272727
Base81: output/input 18.927892607143722 / 15; Ratio: 19 / 15 = 1.2666666666666666
Base82: output/input 23.908573597131127 / 19; Ratio: 24 / 19 = 1.263157894736842
Base85: output/input 4.9926740807112 / 4; Ratio: 5 / 4 = 1.25
Base86: output/input 4.979564524879807 / 4; Ratio: 5 / 4 = 1.25
Base87: output/input 4.966674008644963 / 4; Ratio: 5 / 4 = 1.25
Base88: output/input 4.953996247544582 / 4; Ratio: 5 / 4 = 1.25
Base89: output/input 4.941525209635524 / 4; Ratio: 5 / 4 = 1.25
Base90: output/input 4.929255102536434 / 4; Ratio: 5 / 4 = 1.25
Base91: output/input 4.917180361275656 / 4; Ratio: 5 / 4 = 1.25
Base92: output/input 4.905295636885699 / 4; Ratio: 5 / 4 = 1.25
Base93: output/input 15.904186303494539 / 13; Ratio: 16 / 13 = 1.2307692307692308
Base94: output/input 10.984670683283468 / 9; Ratio: 11 / 9 = 1.2222222222222223

Conclusions

This output provides several interesting insights:

All the "power of two" bases, e.g. Base16/32/64, always have a whole number of required digits, as the source base is also the "power of two"! This simple fact makes it even easier to calculate the optimal groups by using a method of finding LCM (Least Common Multiple), also shown in the previous article.
There are a few groups of adjacent bases that require the same number of digits but are different by the size of their alphabets. It seems reasonable to prefer smaller alphabets, as less special symbols lead to better readability, e.g. when an encoded text needs to be used within a value of some variable in a programming language, or read verbally over a voice channel (encoded license keys).
Usually, the size of binary files, and especially executable files, appears to be evenly divisible by 4. This makes reasonable to use bases, that have 4-byte input groups. Then, there will be fewer chances to convert files, where the last byte group doesn't have all the needed data to perform the conversion. Although, even if it happens, it usually addresses using padding by NULL-symbols. The Base32 and Base64 for padding uses one extra symbol (out of the alphabet) '=', and Ascii85 uses an even smarter approach, with no extra symbols on the output stream.
Among all bases in the list, there is one outstanding base, Base85. It uses 4 input bytes that aligned with the average case of binary files. 5 output bytes give only 25% overhead which provides better efficiency than Base64 (with its 33.3%). Both groups fit CPU's registers all modern computers. All these factors make this encoding much more optimal for a binary-to-text encoding than commonly used nowadays on the Internet encoding - Base64 or some times ago on the FidoNet - UUEncode (which internally is the same Base64). With the differences in alphabets, Base85 is used in PDF, Git, ZeroMQ, and also implemented in the Standard Python Library base64.
There are also known to be used Crockford-Base32, Base36, and Base58 in special applications, as efficiency is not the main consideration for their use and they meet other requirements.

Convert binary data to a text with the lowest overhead

2020-04-18T23:10:29-07:00

TLDR: quick summary of the article

This article discusses binary/text converters, the most popular implementations, and a non-standard approach that uses place-based single-number encoding by representing a file as a large number and then converting it to another large number with any non-256 (1-byte/8-bit) radix. To make it practical, it makes sense to limit a radix (base) to 94 for matching numbers to all possible printable symbols within the 7-bit ASCII table. It is probably a theoretical prototype and has a purely academic flavor, as the time and space complexities make it applicable only to small files (up to a few tens of kilobytes), although it allows one to choose any base with no dependencies on powers of two, e.g. 7 or 77.

Background

The main purpose of such converters is to convert a binary file represented by 256 different symbols (radix-256, 1 byte, 2^8) into a form suitable for transmission over a channel with a limited range of supported symbols. A good example is any text-based network protocol, such as HTTP (before ver. 2) or SMTP, where all transmitted binary data must be reversibly converted to a pure text form without control symbols. As you may know, ASCII codes from 0 to 31 are considered control characters, and therefore will definitely be lost during transmission over any logical channel that doesn't allow endpoints to transmit full 8-bit bytes (binary) with codes from 0 to 255. This limits the number of allowed symbols to less than 224 (256-32), but it's actually limited by the first 128 (2^7, 7 bits) standardized symbols in the ASCII table, and even more.

The standard solution today is the Base64 algorithm defined in RFC 4648 (easy to read and understand). It also describes Base32 and Base16 as possible variants. The key point here is that they all share the same property of being powers of two. The wider the range of supported symbols (codes), the more space-efficient the result of the conversion. It will be larger anyway, the question is how much larger. For example, Base64 encoding gives about 33% larger output, because 3 input bytes (8 valued bits) are translated into 4 output bytes (6 valued bits, 2^6=64). So the ratio is always 4/3, i.e. the output is larger by 1/3 or 33.(3)%. Practically speaking, Base32 is very inefficient because it means translating 5 input bytes (8 valued bits) into 8 output bytes (5 valued bits, 2^5=32) and the ratio is 8/5, i.e. the output is larger by 3/5 or 60%. In this context, it is hard to consider any kind of efficiency of Base16, since its output size is larger by 100% (each byte of 8 valued bits is represented by two bytes of 4 valued bits, also known as nibbles, 2^4=16). By the way, this is a well-used representation of 8-bit bytes, called hexadecimal.

If you're curious how these input/output byte ratios were calculated for the Base64/32/16 encodings, the answer is LCM (Least Common Multiple). Let's calculate it ourselves, and for that we need another function, the GCD (Greatest Common Divisor)

Base64 (Input: 8 bits, Output: 6 bits):
- LCM(8, 6) = 8*6/GCD(8,6) = 24 bit
- Input: 24 / 8 = 3 bytes
- Output: 24 / 6 = 4 bytes
- Ratio (Output/Input): 4/3
Base32 (Input: 8 bits, Output: 5 bits):
- LCM(8, 5) = 8*5/GCD(8,5) = 40 bit
- Input: 40 / 8 = 5 bytes
- Output: 40 / 5 = 8 bytes
- Ratio (Output/Input): 8/5
Base16 (Input: 8 bits, Output: 4 bits):
- LCM(8, 4) = 8*4/GCD(8,4) = 8 bit
- Input: 8 / 8 = 1 byte
- Output: 8 / 4 = 2 bytes
- Ratio (Output/Input): 2/1

The key problem

What if a channel is only capable of transmitting a few different symbols, such as 9 or 17? That is, we have a file represented by a 256-symbol alphabet (a normal 8-bit byte), we are not really limited by computing power or memory constraints on either side, but we are only able to send 7 different symbols instead of 256? Base64/32/16 are not a solution here. Then Base7 is the only possible output format.

Another example, what if the amount of data transmitted is a concern for a channel? Base64, as it has been shown, increases the data by 33%, no matter what is transmitted, always. Base94, for example, only increases the output by 22%.

It may seem that Base94 is not the limit. If the first 32 ASCII codes are control characters, and there are 256 codes in total, what stops you from using an alphabet of 256 - 32 = 224 symbols? It turns out that there is a limit. Not all of the 224 ASCII codes are printable characters or have a standard representation. In general, only 7 bits (0..127) are standardized, and the rest (128..255) is used for the variety of locales, e.g. Koi8-R, Windows-1251, etc. This means that only 128 - 32 = 96 are available in the standardized range. In addition, the ASCII code 32 is the space character, and 127 doesn't have a visible character either. So 96 - 2 gives us the 94 printable characters that have the same association with their codes on most machines.

Solution

Although this solution is quite simple, this simplicity also imposes a significant computational constraint. The entire input file can be treated as a large number with a base of 256. It could easily be a really big number, requiring thousands of bits. Then all we have to do is convert that big number to a different base. That's it. And Python3 makes it even easier! Normally, conversions between different bases are done via an intermediate base10. The good news is that Python3 has built-in support for large number calculations. The int class has a method that reads any number of bytes and automatically represents them as a large Base10 number with a desired endian. So essentially all of this complexity can be implemented in just two lines of code!

with open('inpit_file', 'rb') as f:
    in_data = int.from_bytes(f.read(), 'big')

where in_data is the big Base10 number. That's only two lines, but that's where most of the math happens and most of the time is spent. So now convert it to any other base, as you'd normally do with normal small decimal numbers.

Read more about stream encoding algorithms with a variable base (16, 32, 36, 64, 58, 85, 94) in my next article The zoo of binary-to-text encoding schemes.

Summary

The article discusses converting binary data to text using different encoding schemes like Base64, Base32, Base16 and a non-standard Base94.
To allow transmission over text-based protocols like HTTP and SMTP, standard encoding schemes like Base64 or Base32, represent binary data as powers of two.
Base64 increases a file size by around 33% while Base32 increases it by 60% due to their encoding ratios of input to output bytes.
For practical purposes, the Base94 may be used to match all printable ASCII characters from 0-127.
Base94 is very efficient and increases a file size by only around 22% compared to Base64's 33% increase.
Python allows implementing the approach easily using its large number capabilities and byte conversion methods.
The key steps of described approach are converting the binary file to a large Base10 number, then converting it to the desired output base.

My notes for the "Pragmatic Thinking and Learning" book

2020-01-18T22:01:10-08:00

New Skill Acquisition

Pragmatic Learning Plan

Dreyfus model

Mastering Knowledge

Gaining Experience

How to start learning

Computer Science vs Information Technology

2019-12-20T15:26:50-08:00

If you ever thought about getting a computer-related (graduated) education, you probably came across a variety of similar disciplines, more or less connected to each other, but grouped under two major fields of study: Computer Science (CS) and Information Technology (IT). The latest one sometimes comes in a broader meaning - Information Communications Technology (ICT), and Computer Science, in turn, is highly linked to Electrical Engineering. But what exactly makes them all different?

Briefly, Computer Science creates computer software technologies, Electrical Engineering creates hardware to run this software in an efficient way, while Information Technology uses them later to create Information Systems for storing, processing and transmitting data.

CS is a study of using computation and computer systems for solving real-world problems. Dealing mostly with software, the study includes the theory of computation and computer architecture, design, development, and application of software systems. The most common problems are organized in groups in particular areas, such as Distributed Systems, Artificial Intelligence, Data Science, Programming Languages and Compilers, Algorithms and Data Structures, etc. Summarizing, CS mainly focuses on finding answers to the following questions (by John DeNero, cs61a):

which real-world problems can be solved using computation
how to solve these problems
how to solve them efficiently

The fact that CS is all about software, makes it tightly coupled to Electrical Engineering that deals with hardware and focuses on designing computer systems and electronic devices for running software in the most efficient way.

Unlike CS, IT is a study of using computers to design, build, and operate Information Systems which are used for storing and processing information (data). ICT extends it by applying telecommunications for receiving and transmitting data. It is crucial to notice, that IT apply existing technologies (e.g. hardware, operating systems, systems software, middleware applications, databases, networks) for creating Information Systems. Hence, IT professionals are users of technologies and utilize existing solutions (hardware and software) to create larger systems for solving a specific business need.

Who is an engineer

2019-12-19T17:29:14-08:00

With the coming of the Industrial Age (approx. 1760-1950), an agricultural society transitioned to an economy, based primarily on massive industrial production. It was the time of the rise of specialized educational centers, where people could get deep knowledge in many different fields of science and became either scientists or engineers.

Briefly, the main difference between scientists and engineers is that scientists discover, but engineers invent. That is Engineers, using discoveries of scientists, invent systems, devices, processes, which they design, develop, implement, build, manage, maintain, and improve as different stages of the Engineering process. Engineering is a practical application of scientific knowledge, integrated with business and management. In other words, engineers act as a bridge between science and society by doing inventions for the real world and people.

In the modern time of the Information Age, the role of an engineer has been extended by non-technical skills, as a result of the globalization and spreading of trade relationships across the globe. These are skills such as intellectual (communication, foreign languages, critical thinking), management (time management, self-organization, planning), and standards awareness (tech certifications, best practices).

Algorithm is...

2019-12-15T16:58:11-08:00

Despite the obvious expectation to find some sort of a definition of the term "Algorithm" here, I have to disappoint you, as there isn't any general or well-accepted definition. But, it's not a unique situation! Take mathematics, for example. Although there are plenty of different "definitions" that can be found in the literature, they all are just oversimplified attempts to explain what an algorithm really means.

In general, an algorithm is a way of describing the logic. And that's why it's so hard to cover all possible forms of it in terms of common rules or definitions. Most prominent mathematicians began seriously thinking about computability and what can be computed at the beginning of the 20th century. But it was so hard to generalize all the cases that eventually they had to limit the consideration by functions defined only on the set of Natural numbers.

The most famous works were done by Alan Turing (related to algorithms) and Alonzo Church (related to computable functions). Alan Turing came up with the thesis which basically says, that if a function is computable then it has an algorithm, and if so, then it can be implemented on the Turing machine (TM). In other words, Turing's thesis makes it clear what can be computed and what is needed to get computed.

Turing machine is an abstract system that has a finite set of states and symbols, a few certain operations, and an endless tape (consisted of cells). The behavior of a TM is controlled by a program that defines a state transition and a next tape movement depending on a symbol that was read. Although, there is no a real-world analog of the TM as it is unlikely possible to have infinite memory. So, to get it more realistic, for a real analog of TM, it means two things:

to have enough memory, at least, as much as needed (analog of the tape)
to have a conditional branching, some sort of if/else and goto statements (analog of state transitions)

All algorithms share the same properties:

deterministic (produces the same result for the same input)
discrete (works with discrete data, like texts, integers, rational numbers)
finite (represented by a finite text)

Turing: thesis, machine, completeness

2019-12-15T15:01:47-08:00

Alan Turing is one of the pioneers of the computability theory and logic formalization. He came up with the hypothesis of which algorithms can be implemented and computed by machines (Turing's thesis), created an abstract model of such machine (Turing machine), and described absolutely vital abilities of any system for being able to realize any logic that can be computed (Turing completeness).

Turing's thesis is only one of the existing formal systems in the computability theory. There are also λ-calculus, Markov algorithms, but they all were implemented on the Turing Machine that is used at this time as a general computational model to classify which real-world systems (mostly programming languages) are able to compute mathematical functions or implement algorithms.

All existing computability theories are defined on discrete values, and the domain is the set of Natural numbers.

I prepared several mindmaps to summarize basic ideas and statements:

Turing's thesis:

Turing machine:

Turing completeness:

Organizing Unstructured Data

2019-08-21T17:08:40-07:00

The main, if not the only, purpose of a computer is to compute information. It doesn't always have to be a computation of mathematical formulas. In general, it is a transformation of one piece of information into another. Computers only work with information that can be represented as discrete data. The input and output of a computer engine are always natural numbers or text (a sequence of symbols from a dictionary that correspond to certain natural numbers).

As long as data is unstructured, it's hard to make some sense of it. But once data is given a structured form, it becomes meaningful and suitable for further transformation.

Type

The simplest form of data organization is Type. In general, a data type defines a set of values with certain properties. It usually defines a size in bytes. A primitive data type is an ordered set of bytes. When a variable of a primitive data type has only one value (holds only one piece of information), it's called a scalar and a type - scalar data type. Well-known examples are integer, float, pointer, and char. A collection of primitive (scalar) data types is called an aggregate data type, and it allows multiple values to be stored. This can be a homogeneous collection, where all elements are of the same type, such as an array, a string, or a file. Or it can be heterogeneous, where elements are of different types, such as a structure or a class. The main property is an ordered set of bytes. The internal organization is simple, straightforward, and all actions (e.g. reading or modifying) are performed directly on the data, according to a hardware architecture that defines the byte order in memory (little-/big-endian).

Data Structure

The next level of data abstraction is called Data Structure. It brings more complexity, but also more flexibility to make the right choice between access speed, ability to grow, modification speed, etc. Internally, it's represented by a collection of the scalar or aggregate data types. The main focus is on the details of the internal organization and a set of rules to control that organization. There are two types of data structures that result from a difference in the memory allocation of the underlying elements:

Array Data Structures (static), based on physically contiguous elements in memory, with no gaps between them;
Linked Data Structures (dynamic), based on elements, dynamically allocated in memory and linked in a linear structure using pointers (usually, one or two)

Well-known examples are linked list, hash (dictionary), set, list. These data structures are defined only by their physical organization in memory and a set of rules for data modifications that are performed directly. All internal implementation details are open. The actions performed on the data structures (add, remove, update, etc.) and the ways in which they are used can vary.

Abstract Data Type (ADT)

A higher level of data abstraction is represented by an Abstract Data Type (ADT), which shifts the focus from "how to store data" to "how to work with data". An ADT represents a logical organization, defined mainly by a list of predefined operations (functions) for manipulating data and controlling its consistency. Internally, data can be stored in any data structure or combination thereof. However, these internals are hidden and should not be directly accessible. All interactions with data are done through an interface (operations exposed to users). Most of ADTs share a common set of primitive operations, such as

create - a constructor of a new instance
destroy - a destructor of an existing instance
add, get - the set-get functions for adding and removing elements of an instance
is_empty, size - useful functions for managing existing data in an instance

The most common examples of ADTs are stack and queue. Both of these ADTs can be implemented using either array or linked data structures, and both have specific rules for adding and removing elements. All of these specifics are abstracted as functions, which in turn, perform appropriate actions on internal data. Dividing an ADT into operations and data structures creates an abstraction barrier that allows you to maintain a solid interface with the flexibility to change internals without side effects on the code using that ADT.

Object

A more comprehensive way of abstracting data is represented by Objects. An object can be thought of as a container for a piece of data that has certain properties. Similar to the ADT, this data is not directly accessible (known as encapsulation or isolation), but instead each object has a set of tightly bound methods that can be applied to operate on its data to produce an expected behavior for that object (known as polymorphism). All such methods are really just functions collected under a class. However, they become methods when called to operate on a particular object. Methods can also be inherited from another class, which is called a superclass. Unlike an ADT, an object doesn't represent a particular type of data, but rather a set of attributes, and it behaves as it should when its methods are invoked. Attributes are nothing more than variables of any type (including ADTs). Formally speaking, classes act as specifications of all of the object's attributes and the methods that can be invoked to deal with those attributes.

The Object-Oriented Programming (OOP) paradigm uses objects as the central elements of a program design. At program runtime, each object exists as an instance of a class. The class, in turn, plays a dual role: it defines the behavior (through a set of methods) of all objects instantiated from it, and it declares a prototype of data that will carry some state within the object once it's instantiated. As long as the state is isolated (incapsulated) in the objects, access to that state is organized by communication between the objects via message passing. It's usually implemented by calling a method of an object, which is equivalent to "passing" a message to that object.

This behavior is completely different from the Structured Programming Paradigm, which instead of maintaining a collection of interacting objects with an an embedded state, relies on dividing of a project's code into a sequence of mostly independent tasks (functions) that operate with an externally (to them) stored state.

Number Classification

2019-08-16T12:42:06-07:00

Mathematics is unique. The unique science if everyone could agree that it is Science. But, it's also hard to argue that it is not Art. Math is absolutely certain, except the cases when it is not ("as far as the laws of mathematics refer to reality, they are not certain; and as far as they are certain, they do not refer to reality"). Still having no one general definition, math doesn't even bother to have one opinion on such the fundamental building block as Numbers. Nevertheless, math is an important part of almost every field of science, engineering, and human life.

Here is the most common and well-accepted number classification tree:

It also shouldn't be a surprise to find slight distinctions in the meaning of the same essences in Math and Computer Science (CS):

Natural numbers. In Math, they are meant to be Positive Integers (1, 2, 3, ...), but in CS they are non-negative Integers which include Zero (0, 1, 2, 3 ...)
Mantissa. In Math, it is a fractional part of the logarithm. In CS, it is significant digits of a floating-point number (thus, quite often are used other definitions in this case, like significand and coefficient)

There is a quite related topic in terms of the values which a variable can take on. In mathematics, a variable may be two different types: continuous and discrete:

A variable is continuous when it can take on infinitely many, uncountable values. There is always another value in between two others in a non-empty range, no matter how close they are.
A variable is discrete when there is always a positive minimum distance between two values in a non-empty range. The set of numbers is finite or countably infinite (e.g. Natural numbers)

The understanding of the discreteness is crucial in Computer Science as all real-world computers internally work only with discrete data (which makes it challenging to represent Irrational numbers). All existing computability theories (e.g. Turing thesis, Church thesis) are defined on discrete values, and the domain is the set of Natural numbers.

Vorakl's notes

How to destroy your OS with tar

The mistake

The root cause

What can be done to prevent it?

Conclusion

A few facts about POSIX

How did we get there?

The birth of POSIX

The POSIX spec

Options and Option Groups

Summary

How to sort arrays natively in Bash

Summary

Availability calculation in "nines" notation

Summary

A little mess with function parameters in Python

Summary

Using udp-link to enhance TCP connections stability

Summary

The zoo of binary-to-text encoding schemes

What's wrong with the positional single number encoding?

What's the essence of a positional numeral system?

Where does the overhead come from?

How to calculate a minimal overhead?

How to calculate optimal input and output groups?

Conclusions

Convert binary data to a text with the lowest overhead

Background

The key problem

Solution

Summary

My notes for the "Pragmatic Thinking and Learning" book

New Skill Acquisition

Pragmatic Learning Plan

Dreyfus model

Mastering Knowledge

Gaining Experience

How to start learning

See also

Computer Science vs Information Technology

Who is an engineer

Algorithm is...

Turing: thesis, machine, completeness

Organizing Unstructured Data

Topics

Type

Data Structure

Abstract Data Type (ADT)

Object

Number Classification