Introduction & background information

NOTE: This content is from an internal talk I gave, thus the reason it may read like a presentation

So what is bcachefs?

bcachefs is a next-generation copy-on-write (COW) filesystem (FS) that aims to provide features similar to Btrfs and ZFS, written by Kent Overstreet

• Copy-on-write (COW), with goal of performance being better than other COW filesystems
• Full checksums on everything
• Mult-device, replication , RAID, Caching, Compression, Encryption
• Sub-volumes, snapshots
• Scalable, 50+ TB tested

Why the need for another FS?

According to Kent[1], paraphrased here

The ability to write a C shared library in rust has been around for some time and there is quite a bit of information about the subject available. Some examples:

• Exposing C and Rust APIs: some thoughts from librsvg
• Creating C/C++ APIs in Rust (site removed?)
• Rust Out Your C by Carol (Nichols || Goulding) (youtube video)
• Exporting a GObject C API from Rust code and using it from C, Python, JavaScript and others
• Rust Once, Run Everywhere

All this information is great, but what I was looking for was a simple step-by-step example which also discussed memory handling and didn’t delve into the use of GObjects. I also included an opaque data type, but I’m not 100% sure if my approach is the most correct.

I used to like Python. Like others I enjoy the productivity it offers and the vast and plentiful libraries that exist. However, over time that fondness has turned to loathing.

The thing that should have been apparent to me long ago is that the Python folks don’t appear to care about end users. They seem to have lost touch with the fact that Python is very popular! Each and every time they make core language behavior changes, API changes, and deprecate things, a lot of code has to accommodate. It’s a non-trivial amount of work to keep Python code working. Especially so if you’re trying to support code that has to run across multiple versions spanning many years. The test matrix just keeps on getting bigger. The code hacks to accommodate versions becoming more and more intrusive.

Ideally it would be great if a DBUS server side library provided

1. Fully implements the functionality needed for common interfaces (Properties, ObjectManager, Introspectable) in a sane and easy way and doesn’t require you to manually supply the interface XML.
2. Allows you to register a number of objects simultaneously, so if you have circular references etc.  This avoids race conditions on client.
3. Ability to auto generate signals when object state changes and represent the state of the object separately for each interface on each object.
4. Freeze/Thaw on objects or the whole model to minimize number of signals, but without requiring the user to manually code stuff up for signals.
5. Configurable process/thread model and thread safe.
6. Incrementally and dynamically add/remove an interface to an object without destroying the object and re-creating and while incrementally adding/removing the state as well.
7. Handle object path construction issues, like having an object that needs an object path to another object that doesn’t yet exist.  This is alleviated of you have #8.
8. Ability to create one or more objects without actually registering them with the service, so you can handle issues like #7 easier, especially when coupled with #2, directly needed for #2.  Thus you create 1 or more objects and register them together.
9. Doesn’t require the use of code generating tools.
10. Allow you to have multiple paths/name spaces which refer to the same objects. This would be useful for services that implement functionality in other services without requiring the clients to change.
11. Allows you to remove a dbus object from the model while you are processing a method on it.  eg. client is calling a remove method on the object it wants to remove.

Introduction

Stratis (which includes stratisd as well as stratis-cli), provides ZFS/Btrfs-style features by integrating layers of existing technology: Linux’s devicemapper subsystem, and the XFS filesystem. The stratisd daemon manages collections of block devices, and exports a D-Bus API. The stratis-cli provides a command-line tool which itself uses the D-Bus API to communicate with stratisd.

1. Installation

# dnf install stratisd stratis-cli

2. Start the service

# systemctl start stratisd
# systemctl enable stratisd
Created symlink /etc/systemd/system/sysinit.target.wants/stratisd.service →
/usr/lib/systemd/system/stratisd.service.

3. Locate a block device that is empty. You can use something like lsblk and blkid to locate, eg.

 
# lsblk
    NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    sda      8:0    0   28G  0 disk 
    ├─sda1   8:1    0    1G  0 part /boot
    ├─sda2   8:2    0  2.8G  0 part \[SWAP\]
    └─sda3   8:3    0   15G  0 part /
    sdb      8:16   0    1T  0 disk  

# blkid -p /dev/sda
  /dev/sda: PTUUID="b7168b63" PTTYPE="dos"

Not empty

Debugging gobject reference leaks can be difficult, very difficult according to the official documentation. If you google this subject you will find numerous hits.  A tool called RefDbg was created to address this specific need. It however appears to have lost effectiveness because (taken from the docs):

Beginning with glib 2.6 the default build does not allow functions to
be overridden using LD_PRELOAD (for performance reasons).  This is the
method that refdbg uses to intercept calls to the GObject reference
count related functions and therefore refdbg will not work.  The
solution is to build glib 2.6.x with the '--disable-visibility'
configure flag or use an older version of glib (<= 2.4.x).  Its often
helpful to build a debug version of glib anyways to get more useful
stack back traces.

I actually tried the tool and got the error “LD_PRELOAD function override not working. Need to build glib with –disable-visibility?” , so I kept looking. Another post lead me to investigate using systemtap which seemed promising, so I looked into it further.  This approach eventually got me the information I needed to find and fix reference count problems.  I’m outlining what I did in hopes that it will be beneficial to others as well.  For the record, I used this approach on storaged, but it should work for any code base that uses gobjects. The overall approach is:

Continuous integration  (CI) support in github is a very useful addition.   Not only can you utilize existing services like Travis CI, you can utilize the github API and roll your own, which is exactly what we did for libStorageMgmt. LibStorageMgmt needs to run tests for hardware specific plugins, so we created our own tooling to hook up github and our hardware which is geographically located across the US.  However, shortly after getting all this in place and working it became pretty obvious that we provided a nice attack vector…

While working on lvm-dubstep the question was posed if D-bus could handle the number of changes that could happen in a short period of time, especially PropertiesChanged signals when a large number of logical volumes or physical volumes were present on the system (eg. 120K PVs and 10K+ LVs).  To test this idea I put together a simple server and client which simply tries to send an arbitrary number of signals as fast as it possibly can.  The number settled upon was 10K because during early testing I was running into time-out exceptions when trying to send more in a row.  Initial testing was done using the dbus-python library and even though numbers seemed sufficient, people asked about sd-bus and sd-bus utilizing kdbus, so the experiment was expanded to include these as well.  Source code for the testing is available here.  

The documentation on the gnu libtool manual states:

The following explanation may help to understand the above rules a bit better: consider that there are three possible kinds of reactions from users of your library to changes in a shared library:

1. Programs using the previous version may use the new version as drop-in replacement, and programs using the new version can also work with the previous one. In other words, no recompiling nor relinking is needed. In this case, bump revision only, don’t touch current nor age.
2. Programs using the previous version may use the new version as drop-in replacement, but programs using the new version may use APIs not present in the previous one. In other words, a program linking against the new version may fail with “unresolved symbols” if linking against the old version at runtime: set revision to 0, bump current and age.
3. Programs may need to be changed, recompiled, relinked in order to use the new version. Bump current, set revision and age to 0.

This was confusing to me until I played around with the -version-info value and looked at the library output on my linux development system.

NOTES:

• Updated 4/2/2015 to reflect new project links and updated command line syntax
• Updated 5/20/2021 to reflect new IRC service
• Updated 7/18/2023 removed IRC channel

Abstract This paper discusses some of the advanced features that can be used in modern storage subsystems to improve IT work flows. Being able to manage storage whether it be direct attached, storage area network (SAN) or networked file system is vital. The ability to manage different vendor solutions consistently using the same tools opens a new range of storage related solutions. LibStorageMgmt meets this need.