lp0 on fire

Leafing through CVMFS Catalogs

2025-04-08T21:31:16+00:00

I’m mainly writing this down so I remember how to dig into these bits of CVMFS internals next time¹.

As has happened a couple of times, I discovered today that the campus firewall was blocking one specific file on the CVMFS Stratum 0 and Stratum 1 servers hosted inside the campus network. Fetching the file from inside the network works as expected, but from outside it fails:

[user@stratum1 ~]$ cvmfs_server snapshot repo.example.org
Replicating from catalog at /gtdb
  Processing chunks [8056 registered chunks]: ........
failed to download http://stratum0.example.org/cvmfs/repo.example.org/data/73/2104e02468491fa4e61599c176b16c98a052d4P (15 - host serving data too slowly)
couldn't reach Stratum 0 - please check the network connection
terminate called after throwing an instance of 'ECvmfsException'
  what():  PANIC: /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc9/build/BUILD/cvmfs-2.11.5/cvmfs/swissknife_pull.cc : 286
Download error

Additional testing with curl shows that no bytes are returned for the request and eventually the connection times out:

[user@stratum1 ~]$ curl -sv \
    http://stratum0.example.org/cvmfs/repo.example.org/data/73/2104e02468491fa4e61599c176b16c98a052d4P >/dev/null
*   Trying 192.0.2.42...
* TCP_NODELAY set
* Connected to stratum0.example.org (192.0.2.42) port 80 (#0)
> GET /cvmfs/repo.example.org/data/73/2104e02468491fa4e61599c176b16c98a052d4P HTTP/1.1
> Host: stratum0.example.org
> User-Agent: curl/7.61.1
> Accept: */*
>
  0     0    0     0    0     0      0      0 --:--:--  0:10:13 --:--:--     0* Recv failure: Connection timed out
  0     0    0     0    0     0      0      0 --:--:--  0:10:14 --:--:--     0
* Closing connection 0
curl: (56) Recv failure: Connection timed out

Upon describing the problem for the firewall admins, they sent back a few lines from the log showing the denial: the file type (or “application” in Palo Altese) was detected as flash (as in, Adobe/Shockwave Flash). Why is my CVMFS repo full of public genomic data serving up Adobe Flash?

Well, it’s not. CVMFS is a chunked, content-hash-addressed filesystem. I had recently published the Genome Taxonomy Database and suspected that the file in question (2104e02468491fa4e61599c176b16c98a052d4P) to be a chunk from that rather large database. But it’s best to confirm such suspicions.

To do that, I need to know what file the chunk belongs to.

Locate the revision that added the chunk

There’s probably a more clever way to do this but I started with the mod date on the hash file:

[user@stratum0 ~]$ ls -lh /srv/cvmfs/repo.example.org/data/73/2104e02468491fa4e61599c176b16c98a052d4P
-rw-r--r-- 1 user user 2.4M Apr  4 14:04 /srv/cvmfs/repo.example.org/data/73/2104e02468491fa4e61599c176b16c98a052d4P

This should point to the revision/tag (unless you have a lot of them around that time):

[user@stratum0 ~]$ cvmfs_server tag repo.example.org | grep -B1 '4 Apr 2025'
stuff | 41 | 1 Apr 2025 12:34:05  |        | Stuff
gtdb  │ 42 │ 4 Apr 2025 14:09:38  │        │ GTDB database

Identify files changed in that revision

The diff subcommand shows me what files were added, removed, or changed:

[user@stratum0 ~]$ cvmfs_server diff -s stuff -d gtdb repo.example.org
d(# regular files): 10
d(# symlinks): 0
d(# directories): 1
d(# catalogs): 1
/ modify directory [link-count, timestamp]
/gtdb add directory +4096 bytes
/gtdb/GTDB add file +33731108154 bytes
/gtdb/GTDB_h add file +13565000049 bytes
/gtdb/GTDB.index add file +2601811970 bytes
/gtdb/GTDB.dbtype add file +4 bytes
/gtdb/GTDB.lookup add file +3621260940 bytes
/gtdb/GTDB.source add file +786640 bytes
/gtdb/GTDB.version add file +28 bytes
/gtdb/GTDB_h.index add file +2559366433 bytes
/gtdb/GTDB_mapping add file +1562822676 bytes
/gtdb/.cvmfscatalog add file +0 bytes
/gtdb/GTDB_h.dbtype add file +4 bytes
/gtdb/GTDB_taxonomy add file +9978672 bytes

Locate the catalog for the offending chunk

Knowing what’s changed, now I can locate the catalog corresponding to those changes:

[user@stratum0 ~]$ cvmfs_server list-catalogs -h repo.example.org
218f6ec14c5f8cf2028d9f419e57157f0dbe8d2d /
├─ ae762dea060a55e6d1d075168bbbf8cd440ca6b2 /gtdb
...

Extract the catalog and verify the filename

With the catalog identified, it needs to be extracted:

[user@stratum0 ~]$ cat /srv/cvmfs/repo.example.org/data/ae/762dea060a55e6d1d075168bbbf8cd440ca6b2C \
    | cvmfs_swissknife zpipe -d > /tmp/catalog.sqlite

I can then verify that the chunk belongs to the GTDB file:

[user@stratum0 ~]$ sqlite3 /tmp/catalog.sqlite
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> SELECT c.name
FROM chunks ch
JOIN catalog c
  ON ch.md5path_1 = c.md5path_1 AND ch.md5path_2 = c.md5path_2
WHERE ch.hash = X'ae762dea060a55e6d1d075168bbbf8cd440ca6b2';
GTDB

Check file contents

But why stop there? A little digging suggests that the file magic for a Flash file should be ASCII CWS, FWS, or FLV, so let’s confirm:

[user@stratum0 ~]$ cat /srv/cvmfs/repo.example.org/data/73/2104e02468491fa4e61599c176b16c98a052d4P \
    | cvmfs_swissknife zpipe -d | head -c 3 ; echo
CWS

And there we are, a Flash application, according to Palo Alto. Viewing the rest of the contents of the chunk also confirm that the data after CWS do not resemble Adobe Flash.

I do confess, I let ChatGPT have a crack at it and it gave me some comically wrong answers, including: 1. making up the command cvmfs_find <repo> --catalogs, which it claims is a “helper tool to search metadata, including nested catalogs”, 2. making up the cvmfs_server catalog-chroot subcommand “to mount the catalog structure”, and 3. making up the cvmfs_swissknife cat subcommand to extract catalogs. Oh you wacky LLM and your confidently incorrect assertions. You’d make a great redditor. ↩

Simon Gladman

2022-11-30T01:01:40+00:00

In the 24 or so hours that have passed since I learned of the passing of my friend Simon, a lot of grief, anger, stories, and thoughts have bounced around my head. It’s become so awash that I decided I needed to write something down, if for no other reason than to help process, and now it has become a sort of memorial for him. My version of one, anyway, as this has become deeply personal and I have debated over whether I’d share it publicly. I have shared it with some other folks close to Simon and it seems to have been helpful for them, so I’ve decided to post it. On top of that, I want the world to know what an incredible guy Simon was.

I thought writing this would be hard, and it is - the thoughts are there, in my head for hours as I’ve had a long drive to make and thus too much time inside my own head, but I don’t know if I can vocalize them well. But in a way, it’s also not hard, because there is just so much you can say about Simon. But I write and then I edit, and write more, edit more, and I don’t know when it’s done. I don’t want it to be done.

One of those thoughts I’d had was, how would I title a memorial to Simon? “Have I told you about my watch?” was one of the first things that came to mind. So it brought both a smile and a welling of tears upon opening the memorial that Björn Grüning wrote for Galaxy only to discover that he had titled his the same. Sure, it’s fresh in both our respective memories, but it’s also a perfect introduction of the person Simon was.

Simon had a layover in Dubai on his way to the 2022 European Galaxy Days. This meant that he had the time to look at some of the airport shops, and, unsurprisingly, a number of watches struck his eye. Simon loved watches, especially fancy Seiko watches, and he came away with one of them. His delightful nerdish excitement to tell everyone and anyone about his new watch became his running joke for the trip. “Have I told you about my watch?” he’d asked me (and many others) with more than a little bit of amusement many times by the end of the week and a half we spent together.

Simon was one of those once in a lifetime individuals who could (and did) befriend anyone. He had a disarming charm that just flowed naturally, but he was genuinely interested in everyone and everything around him. He was the one who brought everyone together, to collaborate, or to play games, or just to sit and talk and have a great time. That nerdish excitement over his watch was how he was about everything, and it was infectious. I am not a watch person but I could listen to Simon talk about his watch, or anything else, all day. His excitement over his favorite band, King Gizzard and the Lizard Wizard (from his hometown of Melbourne, no less!) led me to check them out, and before long they were my favorite band too. He told stories about his past adventures, both ours together and his with others, with the same exuberance. He had a wonderful memory for such things, and I have the opposite, which meant that reliving those times through his stories brought back many fantastic memories.

The circumstances of my life have dwindled my friends list significantly over my middle aged years. It has become rare for me to see my few remaining local friends. In some ways, a friendship with Simon was easier - there was no guilt over the inability to see each other because we were on opposite sides of the world. So we had our friendship online, mainly in a group chat with Helena, punctuated by the brief times once or so per year that we could be together in the real world.

Our chats were a great comfort to me. With Simon in Australia and Helena in Europe, we were almost evenly distributed across time zones. I’d wake up to a log full of messages between the two of them to read, maybe catching Simon briefly before he was off to bed. Helena and I would chat all day, and then toward the end of my day, Simon might reappear to start his day and catch up on the chat that he had missed. It was asynchronous and wonky but it worked well enough for the three of us, as we always had the anticipation of our next face-to-face meeting to keep us going. We had plans for a week-long get-together in State College before the 2020 GCC in Toronto, only to have those plans crushed by COVID. So too were the plans for hanging out in Belgium and the Netherlands after the 2021 GCC in Ghent (in Freiburg we had discussed the possibility of a consolation prize: a Galaxy Admin Training in Ghent). When we finally did get to reunite in Minneapolis and Freiburg in 2022, Simon’s hugs were long and crushing, and signified the truth of how difficult those years apart had been, and how good it was to finally see eachother again.

Simon and I were together on a fateful night in November 2016, in Salt Lake City, after teaching a day of the very first Galaxy Admin Training. We went to a bar to watch the election results pour in, and stared with increasing dismay as what seemed like a probably ok thing earlier in the day turned into a nightmare. I left early in a funk; the next morning I met Simon at the tram stop and he cheered me up. He was always able to do that, he was a good listener and commiserate, but he could steer things to a positive place in a respectful but effortless manner. In online chats, he was always looking ahead, so excited for the next thing and the next time we’d be together.

He cared deeply about his friends and would talk them up to anyone. I know more about his friend Maddi than I do about some of my own friends, and I’ve never even met her. But Simon was so proud of her and her accomplishments that he couldn’t help but talk about her the way a father talks excitedly about his child. He was unquestionably genuine in his feelings.

As we walked around Freiburg this past October, Simon told me how he’d love to spend a year working abroad as his kids were reaching the age of independence and self-sufficiency. I think he was seriously considering it. He had gotten to a relatively good place in life, in a happy relationship with a wonderful woman, two incredible kids that he loved dearly and spoke about proudly and endlessly, and a job that he enjoyed and that afforded him opportunities to travel around the world and make connections that truly seemed to mean everything to him.

I used to imagine how my own life might go as I reached the same milestones. I imagined taking trips every year or so to visit my two improbable friends, a Texan in Europe and an Australian. Simon was ten years older than me, Helena is ten years younger, and yet somehow we all clicked, despite varying personal interests and lifestyles. I imagined a time 20 years from now, after Simon had retired and I no longer had the opportunity to meet up with him at conferences - we’d see him at home in Australia, or plan trips together somewhere else.

When I visited Melbourne in 2017 for the Galaxy Australasia Meeting, I had one extra day, and Simon drove us down the breathtaking Great Ocean Road. I was reluctant to get in the water with Australia’s notoriously deadly ocean creatures, but he reassured me it’d be fine, and I had a wonderful time. He was already planning us an extended trip down the coast after the 2023 GCC, which in a great cruel irony, will finally come to Australia.

The 2019 Galaxy Admin Training was held at Penn State in January/February. Simon and Helena came and experienced one of the coldest weeks in recorded State College history, reaching temps of -20°F. I picked Simon up at the airport and - in a story he loved to retell - he pulled a light puffer jacket out of a small stuff sack to show he’d prepared for this winter trip. I just looked at him and shook my head. He and Helena both ended up borrowing my old coats for the duration as neither of them had a need for such heavy coats at home. I drove my friends up to one of the scenic overlooks over State College and we hurriedly took some photos before retreating back to the warmth of the car. Helena got “proper” Tex-Mex for the first time since leaving the US. Both got to try the abomination known as a Pittsburgh-style sandwich.

In addition to learning all of the unpleasantness that comes from being outdoors in horribly cold temperatures, Simon also got to experience some of the fun. I took him up to a nearby state park, where some folks were clearing the snow off a frozen pond to play hockey, and Simon had a blast sliding around on ice for the first time in his life.

⁦@SimonGladman1⁩ discovered sding on a frozen pond pic.twitter.com/604YD1vCTi
— Nate Coraor (@natefoo@hachyderm.io) (@natefoo) February 2, 2019

Later in the week we went ice skating (indoors thankfully!), and tried to go snow tubing but the temperature had swung too warm, so we got coffee at Rothrock instead. Simon loved coffee more than anyone I know, and I could always count on him to have already found a good coffee shop near every conference venue before I even arrived. More recently, we’d gotten into the practice of trading beans every time we got together. I had worried about how I’d keep us all entertained that week, but my worries ended up being silly in hindsight; Simon was happy just being together, no matter what we did, and I realized that I was too.

The circumstances of my life have meant that spending any significant extra time around conferences had been impossible, until this October, when I was finally able to stay almost a week extra after the European Galaxy Days, and I will be forever grateful that I did. On Sunday after the conference, Simon and I rode the train up to Basel. We had no plans and no expectations due to Germany’s (and Switzerland’s) famed lack of Sunday openness, but I ended up having one of the best days I can remember, just wandering the old city of Basel together. We walked around inside the historic Münster and ultimately decided to take the climb up its 62.7m high Martinsturm spire. Simon and I squeezed our portly frames through the tiny spaces made for the tiny people of the Middle Ages; no small feat for either of us, but much trickier for him at his towering height. We marveled at the sights over the city and he gave me his assessment of a fire he saw off in the distance - Simon was a volunteer firefighter and loved to talk about that part of his life as much as I, completely foreign to the subject, loved to hear about it.

He was a bit appalled to learn that our vantage point was in fact only the first stop - around a narrow ledge with a low wall, another door took us much further up the tower, to even more marvelous (if not a bit terrifying) views. After retreating, with feet firmly back on cobblestone, we lucked into an idyllic lunch of Rösti with a beer outside on the Münsterplatz at one of the only restaurants open on Sunday. I’ll remember that day for the rest of my life.

Simon’s partner Tamara reached out to me secretly in advance of his 50th birthday, which was sadly occurring amidst COVID isolation and lockdowns in Melbourne. She was compiling a video of all of his friends wishing him a happy birthday, and asked if I could send the call out to all his friends in the Galaxy community. I racked my brain to think who all those people might be and felt guilty over who I might be leaving off the list because I didn’t know if they knew him that well, but it’s clear now I could have just sent it to everyone. As his good friend and colleague Andrew Lonie so perfectly put it:

Simon was a people person first and foremost and made friends not just easily, but by default - if he met you, he liked you and you liked him back - that’s just the way it was.

All the people whose lives Simon enriched are at a deep loss, and most of all his family, his children and partner who he loved dearly. I miss you, Simon.

The Ansible Playbook you build isn’t the Ansible Playbook you run

2019-01-29T04:39:25+00:00

The Mistake

A common practice of mine in developing Ansible playbooks is to write a few pieces (add a role, write some tasks, etc.), run the playbook, add some more, run again, and so on. At the end, I have a working playbook that I can apply to other hosts - or so I think.

The mistake is that running that playbook on another target runs the whole playbook at once, not incrementally as I did when I was writing it. This means that unless I only ever add new tasks and roles after the ones I’ve already run, and I never go back and make changes to earlier ones, the order of tasks will be different, but that might not be obvious. Here’s an example.

Example

This week some colleagues and I are teaching the 2019 Galaxy Admin Training at Penn State. For this training, we decided to do away with command line editing as much as possible (although a student may choose not to use Ansible when they get home, the methods taught here apply to deployment and configuration management in general). This means we (which is to say, mostly Helena) developed a training exercise for deploying Galaxy with Ansible.

As I do in development, we had the students build their playbooks step by step. At the end of the day, in preparation for the next day, I ran the playbook on all the unused training cloud instances so that they’d be up to date, if anyone needed to switch instances. That’s where the problems began:

TASK [Create the Galaxy FTP upload dir]
******************************************************************************
fatal: [jan-2019-training-inst-3]: FAILED! => {
    "changed": false,
    "msg": "chown failed: failed to look up user galaxy",
}

To understand what’s happened, have a look at the playbook:

- hosts: galaxyservers
  pre_tasks:
    - name: Create the Galaxy FTP upload dir
      file:
        path: "/srv/galaxy/ftp"
        owner: galaxy
        group: galaxy
        mode: 0750
        state: directory
  roles:
    - galaxyproject.repos
    - galaxyproject.postgresql
    - role: natefoo.postgresql_objects
      become: true
      become_user: postgres
    - galaxyproject.galaxy
    - usegalaxy-eu.supervisor
    - geerlingguy.nginx
    - galaxyproject.proftpd

When building the playbook, we added the FTP dir creation pre_task when adding the galaxyproject.proftpd role (and we added the roles in the order seen, running the playbook between each one). The galaxyproject.galaxy role creates the galaxy user. So on a fresh target system, the pre_task no longer works, since we now run it before the role that creates the user, when the first time around, we ran it after.

Running it as a post_task doesn’t work either. Galaxy starts when usegalaxy-eu.supervisor’s handlers fire, at the end of all the role executions, before any tasks or post_tasks run. And when that happens, you get:

RUNNING HANDLER [Restart Galaxy] *********************************************
fatal: [jan-2019-training-inst-3]: FAILED! => {
     "changed": false,
     "msg": "galaxy: stopped\ngalaxy: ERROR (spawn error)\n"
}

And the error is:

ubuntu@jan-2019-training-inst-3:~$ sudo supervisorctl tail galaxy stderr
f2f0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 306752 bytes (299 KB) for 4 cores
created farm 1 name: job-handlers mules:1,2
*** Operational MODE: threaded ***
added /srv/galaxy/server/lib/ to pythonpath.
DEBUG:galaxy.app:python path is: /srv/galaxy/server/lib/, ., , /srv/galaxy/venv/bin, /srv/galaxy/venv/lib/python2.7, /srv/galaxy/venv/lib/python2.7/plat-x86_64-linux-gnu, /srv/galaxy/venv/lib/python2.7/lib-tk, /srv/galaxy/venv/lib/python2.7/lib-old, /srv/galaxy/venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /srv/galaxy/venv/local/lib/python2.7/site-packages, /srv/galaxy/venv/lib/python2.7/site-packages
DEBUG:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/webapps/galaxy/buildapp.py", line 49, in app_factory
    app = galaxy.app.UniverseApplication(global_conf=global_conf, **kwargs)
  File "/srv/galaxy/server/lib/galaxy/app.py", line 67, in __init__
    self.config.check()
  File "/srv/galaxy/server/lib/galaxy/config.py", line 830, in check
    self._ensure_directory(path)
  File "/srv/galaxy/server/lib/galaxy/config.py", line 815, in _ensure_directory
    raise ConfigurationError("Unable to create missing directory: %s\n%s" % (path, e))
ConfigurationError: Unable to create missing directory: /srv/galaxy/ftp
[Errno 13] Permission denied: '/srv/galaxy/ftp'

We’re running Galaxy with privilege separation, so /srv/galaxy is owned by root. This failure is the reason we added the pre_task in the first place.

The solution in this case was relatively simple: galaxyproject.proftpd has an option to create the FTP directory, which I enabled. If not, however, this would be difficult to fix. There’s no way to insert tasks between roles unless you run all of your roles from tasks using include_role, which is a hack. Assuming I don’t want to modify roles I’ve fetched from Ansible Galaxy (no relation) to create the dir, the next best solution would be to create a single-task role to run after galaxyproject.galaxy to create the FTP directory. Not ideal.

Discussion

There are really two problems here, as far as I’m concerned.

Ansible doesn’t have a way to specify that tasks can fail and should be retried at a later time. I actually came to Ansible from CFEngine which (it’s been a few years, but as I recall) ran in passes - actions it attempted could fail, but it’d run other actions anyway and retry the failed ones later. Ansible has no such mechanism, its execution is pretty much linear. The fact that you can’t insert a task between roles is really just an annoyance that makes it difficult to work around this larger complaint.
I try to write the world’s most generalized and unopinionated roles, but Ansible does not make this easy. Creating directories isn’t always as simple as you might think. Naïve roles will just use a become or assume they are running with the appropriate privileges, but neither of these are appropriate for a role that’s going to be used by others, and could downright fail if the directory is on, for example, NFS with root squashing. The galaxyproject.proftpd’s option to create the FTP directory is disabled by default for this reason (indeed, I myself can’t use it for usegalaxy.org). This is why I end up writing a lot of my own roles, and the reason they end up so complex.

There is no construct in Ansible for a role to assert that it needs certain privileges, and for the consumer of the role to configure the method by which those privileges should be obtained. You run a role as either the user you logged in as, or with become, but that’s too limiting. The galaxyproject.galaxy role, for example, needs to create some files as one user and some as another, and execute commands or perform actions as both of those users. If it can’t do that, you either have to:
1. Run everything as one user, this is less secure,
2. Use “enable/disable” variables and a bunch of whens on a bunch of include_tasks’s to break the role up by privilege, then instruct role consumers on how to run the role multiple times with the correct combination of remote_user or become/become_user and the enable/disable vars set as role vars, or
3. Some kind of nutty kludgery to actually provide the missing assert/configure functionalty.

Somewhat related, but Ansible does not make it easy to see what privileges it’s obtaining and how it’s obtaining them. You can see most of this with -vvv but it’s not easy: The ssh user is displayed on the first line after the task header, but only if ansible_user/remote_user is set on some level. The become_user is buried in the module execution command lines.

Conclusion

My complaints in the discussion aside, the takeaway is that simply building a playbook that works is not sufficient, it needs to be run again from scratch. In addition, any time you’re adding to it later, always consider whether the tasks you’re adding depend on other things set up by the playbook. If they are, further consider whether those dependencies run earlier or later in playbook. Forgetting this consideration is a trap I’ve fallen in to many times.

Link Aggregations and Jumbo Frames in SmartOS

2017-11-09T05:25:23+00:00

Background

For a long while now I’ve been wanting to enable 802.3ad Link Aggregation (LACP) in conjunction with jumbo frames on my SmartOS servers. This is a setup I’d used on Solaris 10, OpenSolaris, and Linux fileservers for years. Unfortunately, due to SmartOS’ unique installation and boot paradigm (where the OS is not installed locally, it is extracted into a ramdisk from the boot archive at boot), getting it set up was not so easy. In fact, back when I set up my first SmartOS server in 2013, it wasn’t possible.

That was rectified and documented (thanks @rmustacc!) a number of years ago, but the one subsequent attmept to use it still ended in failure, so I gave up. Fast forward to yesterday, when I was standing up my first new SmartOS server in a couple years and decided to give aggr+jumbos another shot.

Trying it out

The documentation isn’t exactly clear on how to combine both options - aggregate interfaces are set up by hand in the SmartOS /usbkey/config config file, but MTUs are set using nictagadm(1M), which also appears to set up aggregate interfaces. After looking through the contents of nictagadm (it’s a shell script) and testing it out, it appears that it’s just a handy tool for setting the MTU on the correct entities in /usbkey/config.

With the config in what was hopefully the correct state, I rebooted and prayed. Unfortunately, the network (svc:/network/physical:default) failed to start. Thankfully, SmartOS’s custom SMF method for the network is very chatty:

# cat /var/svc/log/network-physical:default.log
[ Nov  8 20:45:25 Executing start method ("/lib/svc/method/net-physical"). ]
[ Nov  8 20:45:25 Timeout override by svc.startd.  Using infinite timeout. ]
+ smf_configure_ip
+ /sbin/zonename -t
+ [ global = global -o shared = exclusive ]
+ return 0
+ LD_LIBRARY_PATH=/lib
+ export LD_LIBRARY_PATH
+ ADMIN_DHCP_TIMEOUT=300
+ ActiveAggrLinks=''
+ typeset -A ActiveAggrLinks
+ smf_netstrategy
+ smf_is_nonglobalzone
+ [ global != global ]
+ return 1
+ /sbin/netstrategy
+ set -- ufs none none
+ [ 0 -eq 0 ]
+ [ ufs = nfs ]
+ _INIT_NET_STRATEGY=none
+ export _INIT_NET_STRATEGY
+ typeset -A plumbedifs
+ smf_is_globalzone
+ [ global = global ]
+ return 0
+ /usr/sbin/dladm init-phys
+ log_if_state before
+ echo '== debug start: before =='
== debug start: before ==
+ /usr/sbin/dladm show-phys
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
bnx0         Ethernet             unknown    0      unknown   bnx0
bnx1         Ethernet             unknown    0      unknown   bnx1
+ /sbin/ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128 
+ echo '== debug end: before =='
== debug end: before ==
+ load_sdc_sysinfo
+ boot_file_config_enabled
+ load_sdc_config
+ load_sdc_bootparams
+ sed -e 's/,/ /g'
+ echo ''
+ create_aggrs
+ typeset links macs mode mtu
+ [[ -z aggr0 ]]
+ aggrs=( aggr0 )
+ eval links='${SYSINFO_Aggregation_aggr0_Interfaces}'
+ links=bnx0,bnx1
+ eval macs='${SYSINFO_Aggregation_aggr0_MACs}'
+ macs=78:2b:cb:1b:06:9d,78:2b:cb:1b:06:9e
+ eval mode='${SYSINFO_Aggregation_aggr0_LACP_mode}'
+ mode=passive
+ eval mtu='${CONFIG_aggr0_mtu}'
+ mtu=9000
+ [[ -z off ]]
+ echo 'Creating aggr: aggr0 (mode=passive, links=bnx0,bnx1)'
Creating aggr: aggr0 (mode=passive, links=bnx0,bnx1)
+ dladm create-aggr -l bnx0 -l bnx1 -L passive aggr0
+ [[ 0 -eq 0 ]]
+ add_active_aggr_links aggr0 78:2b:cb:1b:06:9d,78:2b:cb:1b:06:9e
+ typeset alink
+ ActiveAggrLinks[78:2b:cb:1b:06:9d]=aggr0
+ ActiveAggrLinks[78:2b:cb:1b:06:9e]=aggr0
+ [[ -n 9000 ]]
+ dladm set-linkprop -p mtu=9000 aggr0
dladm: warning: cannot set link property 'mtu' on 'aggr0': operation not supported
+ [[ 1 -ne 0 ]]
+ echo 'Failed to set mtu on aggr aggr0 to 9000'
Failed to set mtu on aggr aggr0 to 9000
+ exit 95
[ Nov  8 20:45:35 Method "start" exited with status 95. ]

What’s going on here?:

dladm: warning: cannot set link property 'mtu' on 'aggr0': operation not supported

This was the very problem I’d had lo those many years ago, which, at the time, I’d attributed to a need to be able to set the MTU on the individual interfaces before the aggegation, e.g.:

# dladm set-linkprop -p mtu=9000 bnx0
# dladm set-linkprop -p mtu=9000 bnx1
# dladm create-aggr -l bnx0 -l bnx1 -L passive aggr0
# dladm set-linkprop -p mtu=9000 aggr0

Well, my conclusion as to the source of the error was completely wrong. As it turns out, the reason set-linkprop fails on the aggregation is because the Broadcom NetXtreme II bnx driver doesn’t support runtime configuration of the physical MTU limit. It can be set on IP interfaces, but not with dladm set-linkprop.

Configuring bnx

The MTU must be configured through the bnx driver’s config file, bnx.conf. The version that ships with the OS can be found in /kernel/drv/bnx.conf. Some sources online and the Solaris 11 docs claim you need to set the Jumbo=... option, but in the case of the driver version currently shipping in illumos, the option name is mtu.

But this is SmartOS, I can’t just change bnx.conf and reboot like I would with traditional Solaris. Thankfully, I stumbled across a solution posted by @nshalman. This clever enhancement by @wesolows and @JohnSonnenschein allows arbitrary files to be added to the boot archive ramdisk via the bootloader. As @nshalman says, @wesolows’ blog post where it’s mentioned is pretty enlightening, as is the “module” syntax documentation.

My hosts boot from PXE, and use iPXE as their bootloader. Thankfully, iPXE’s module (imgfetch) command works in exactly the same way as the GRUB equivalent, and it pulls the file off my TFTP server just like the rest of the boot components. Over on my (Debian) boot server, I now have:

# cat /var/lib/tftpboot/smartos/bnx.conf
mtu=9000,9000,9000,9000,9000,9000,9000,9000,9000,9000,9000,9000,9000,9000,9000,9000;

And my menu.ipxe entry for booting SmartOS has changed from:

kernel /smartos/${smartos-version}/platform/i86pc/kernel/amd64/unix -B smartos=true,console=${smartos-console},${smartos-console}-mode="115200,8,n,1,-" -v
initrd /smartos/${smartos-version}/platform/i86pc/amd64/boot_archive
boot

To:

kernel /smartos/${smartos-version}/platform/i86pc/kernel/amd64/unix -B smartos=true,console=${smartos-console},${smartos-console}-mode="115200,8,n,1,-" -v
module /smartos/${smartos-version}/platform/i86pc/amd64/boot_archive type=rootfs name=ramdisk
module /smartos/bnx.conf type=file name=kernel/drv/bnx.conf
boot

Regarding the name argument to the type=rootfs module, the documentation claims “This option is ignored for rootfs modules”. However, both @wesolows and @nshalman’s examples use it, and I didn’t test without it, so I left it alone.

Reboot, and you should find that bnx.conf has been copied to /system/boot/kernel/drv/bnx.conf. At first I was looking for it in /kernel/drv/bnx.conf, and thought it must not be working, but the documentation clears this point up.

I also thought it wasn’t working because even with bnx.conf in place and disabling all network configuration in /usbkey/config, dladm set-linkprop -p mtu=9000 bnx0 was still failing. It turns out that this is normal for the bnx driver. The MTU cannot be set via set-linkprop, it must be done through bnx.conf. Thankfully, once you have the aggregation configured correctly, the call to dladm set-linkprop -p mtu=9000 aggr0 in /lib/svc/method/net-physical does not fail.

Here is my working /usbkey/config:

#
# This file was auto-generated and must be source-able by bash.
#

aggr0_aggr=78:2b:cb:1b:06:9d,78:2b:cb:1b:6:9e
aggr0_lacp_mode=passive
aggr0_mtu=9000

admin_nic=aggr0
admin_mtu=9000
admin_ip=172.18.2.13
admin_netmask=255.255.255.0
admin_gateway=172.18.2.1

external_nic=aggr0
external_mtu=9000
external0_ip=128.118.250.13
external0_netmask=255.255.255.224
external0_gateway=128.118.250.1
external0_vlan_id=306

headnode_default_gateway=128.118.250.1

dns_resolvers=128.118.250.8,54.172.110.33
dns_domain=galaxyproject.org

ntp_hosts=0.smartos.pool.ntp.org
compute_node_ntp_hosts=dhcp

hostname=westmalle.galaxyproject.org

Conclusion

The main problem here seemed to be (other than my confusion at various points) the bnx driver. Many people report having an overall better experience with Intel cards, which I do not doubt.

So, I’m happy to report that LACP + jumbo frames are working great (and probably have been for years), as long as you can thwack your network device’s driver in to submission.

Thanks as always to the Joyeurs for their hard work and support.

Addendum 2017-11-09 11:51:00 -0400: I’ve disabled aggr0_lacp_mode and set my switch to use non-LACP trunking. Since I boot from PXE, and because LACP is not activated until the OS is booted, PXE was unable to contact the network to boot. Multilink trunking/failover still works and I wasn’t using dynamic LACP anyway, so this is not a huge issue.

Debian root in a btrfs subvolume

2015-11-04T15:09:49+00:00

Background

If you select btrfs as your filesystem of choice in the Ubuntu 14.04 installer, it’ll create btrfs subvolumes for any “partitions” you request. These partitions will be subvolumes named @ for /, @home for /home, etc.

Sadly, while the Debian installer will install into btrfs, it will not use subvolumes. Thus, after installing Debian, if you want to get your root filesystem into a subvolume, you have to do a bit of work. I wanted to do this because it makes things like snapshots much cleaner when the OS is not installed directly in the btrfs root (otherwise you can’t have any space in btrfs “outside” the mounted filesystem).

Process

Install Debian as normal.
Once installed, I like to immediately create a snapshot of the installed system:

# btrfs subvol snap -r / /@stretch_fcs

Create the snapshot that will become the new root fs:

# btrfs subvol snap /@stretch_fcs /@stretch

Update grub config in the snapshot to refer to the subvolume (thanks goncalopp@stackexchange). My procedure is slightly different since I’m using EFI boot (mounting /boot/efi in the chroot is probably not even necessary, but, just in case). If your /boot is a separate partition, you would still need to mount /boot before /boot/efi:

# btrfs mount -o subvol=@stretch /mnt
# cd /mnt
# mount -o bind /dev dev
# mount -o bind /sys sys
# mount -o bind /proc proc
# mount -o bind /boot/efi boot/efi
# chroot .
# update-grub
# exit

Update the subvolume’s /etc/fstab to mount the subvolume as the root filesystem. Excuse the lazy regexes, you probably want to just edit /etc/fstab by hand, but I don’t like putting “edit this file” in instructions):

# grep '^[^#].* / ' /etc/fstab
UUID=71d1dc27-36d5-492b-b51c-f852fdaa6b4e /               btrfs   defaults        0       1
# sed -ie '/ \/ /s#defaults\([^ ]*\) *#defaults\1,subvol=@stretch #' /tmp/fstab
# grep '^[^#].* / ' /etc/fstab
UUID=71d1dc27-36d5-492b-b51c-f852fdaa6b4e /               btrfs   defaults,subvol=@stretch 0       1

Set the default btrfs filesystem to the @stretch snapshot. We will undo this later because leaving the subvolume as the default means that it’s more cumbersome to mount the btrfs root. However, if we were to reboot now without changing the default, Grub would boot using /boot/grub/grub.cfg rather than /@stretch/boot/grub/grub.cfg, and / would be the mounted filesystem once booted. (Aside, btrfs can’t select a subvolume by name and just print its id? Annoying):

# btrfs subvol set-default $(btrfs subvol list / | awk '$NF == "@stretch" { print $2 }') /

Reboot:

# reboot

Verify that you have booted with @stretch as the root filesystem:

/
	Name: 			@stretch
	uuid: 			284d7ea5-f406-2541-953f-60772fc247af
	Parent uuid: 		09184ee6-f5f9-5a4c-9d1b-2557b0541f0a
	Creation time: 		2015-11-04 10:04:10
	Object ID: 		282
	Generation (Gen): 	502
	Gen at creation: 	482
	Parent: 		5
	Top Level: 		5
	Flags: 			-
	Snapshot(s):

Reset / as the default subvolume (the btrfs root’s subvolume id is always 5):

# btrfs subvol set-default 5 /

Reinstall grub so that it knows to read its config from /@stretch/boot/grub/grub.cfg (assuming your boot device is /dev/sda):

# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.2.0-1-amd64
Found initrd image: /boot/initrd.img-4.2.0-1-amd64
done
# update-grub /dev/sda
Installing for x86_64-efi platform.
Installation finished. No error reported.

Reboot:

# reboot

You should now be able to mount /dev/sda3 (or whatever your btrfs device is) and see the entire tree. I just put it in /etc/fstab and keep it mounted at /btrfs:

# grep /btrfs /etc/fstab
UUID=71d1dc27-36d5-492b-b51c-f852fdaa6b4e /btrfs          btrfs   defaults      0       1
# mkdir /btrfs && mount /btrfs
# ls -l /btrfs
bin   dev  home        lib    media  opt   root  sbin  @stretch      sys  usr  vmlinuz
boot  etc  initrd.img  lib64  mnt    proc  run   srv   @stretch_fcs  tmp  var

If you want, you can now remove <btrfs_root>/<non-subvol-contents> e.g.:

# cd /btrfs
# rm -rf bin boot dev etc home initrd.img lib lib64 media mnt opt proc root run sbin srv sys tmp usr var vmlinuz

Now /btrfs is free for you to create things in, e.g. a snapshots/ directory in which you can create your btrfs snapshots.

Conclusion

TODO

My desktop originally ran Ubuntu 14.04. I later wanted to switch to Debian Stretch without wiping out Ubuntu. I can now dual boot between them thanks to btrfs subvolumes and UEFI. So I need to document that…

Python Packaging: What’s the Big Wheel?

2015-08-28T20:03:10+00:00

Background

Python modules have numerous binary packaging formats, but the one that’s emerged as the winner has been the Wheel format. Wheel comes with a few improvements over Python’s previous binary packaging format, Eggs. Unfortunately, it does not solve the largest problem that we with eggs. Armin Ronacher explains this quite well:

There are a few problems with wheels however. One of the problems is that wheels inherit some of the problems that egg already had. For instance Linux binary distributions are still not an option for most people because of two basic problems: Python itself being compiled in different forms on Linux and modules being linked against different system libraries. The first problem is caused by Python 2 coming in two flavours that are both incompatible to each other: UCS2 Pythons and UCS4 Pythons. Depending on which mode Python is compiled with the ABI looks different. Presently the wheel format (from what I can tell) does not annotate for which Python unicode mode a library is linked. A separate problem is that Linux distributions are less compatible to each other as you would wish and concerns have been brought up that wheels compiled on one distribution will not work on others.

The end effect of this is that you presently cannot upload binary wheels to PyPI on concerns of incompatibility with different setups.

Here’s why that’s such a big problem for Galaxy:

Galaxy has a huge (and ever-growing) list of dependent python modules with C extensions. If we did not prebuild and distribute eggs for these, the initial setup to get Galaxy running would be long and problematic. Some people who download Galaxy to develop tools may not even have compilers installed, let alone the multitude of -dev or -devel packages that aren’t part of a default Debian or RHEL installation that would be required to build all of these packages from source. One of the things that I feel makes Galaxy so accessible is that you can start using it immediately after you clone the source. So that ability to clone and start and have it work as reliably (and quickly) as possible is a high priority.

Mac OS X wheels are allowed in PyPI, and that’s great as a lot of development happens on OS X, but almost all production Galaxy servers run on Linux, and for this, we still have no solution.

For a long time in Galaxy, we’ve monkeypatched setuptools’ platform detection to add the necessary UCS2/UCS4 detection and left it at that, hoping a better solution would come along. When Wheels emerged as the new packaging format but did not solve these problems, I decided to put in the work to solve them.

Preliminary Work and Discussion

This work began as some discussions with PyPA people on IRC but after making a quick implementation that turned out to be too naive, I opened a discussion with the Distutils-SIG to figure out where to go with it.

A Good Tangent

This discussion quickly grew in scope to discuss another failing of wheel, in that it does not provide a mechanism for dealing with externally linked dependencies. My canonical example here is the psycopg2 Python module, the most popular DB-API implementation for PostgreSQL. It depends on the libpq (PostgreSQL’s C API) shared library being available at runtime. So even if we do build and host a psycopg2 wheel, anyone wanting to use it has to install whatever package provides (or compile from source) libpq.so on their system and ensure it’s on the runtime linker path, or else psycopg2 will fail to import. Wheels provide no mechanism to inform what system libraries, packages, or anything else that’s external what they depend on. A variety of solutions were proposed, check out the thread if you’d like to see some of the very good thinking on this subject.

Of course, this is a general wheel problem, not just wheels on Linux - so if a psycopg2 wheel for OS X was in PyPI today, and you installed it but didn’t have libpq installed, you’d run into the same problem you’d have on Linux.

This problem interests me (for example, to solve it for Galaxy’s eggs, we compile a static version of libpq and link that into the egg), but it isn’t the problem I’m trying to solve (and indeed, if you follow the discussion far enough, it eventually finds its own thread).

The State of Things

Solving the external dependency problem is good and worth doing, but for the moment I want to improve these problems in the current state:

Linux wheels are not allowed in PyPI.
Compiling a package from a source distribution requires installation of a variety of development tools and header packages.

Figuring out the right mix of packages may require multiple iterations of attempted package installs, with errors that may be fairly confusing for people not used to compiling software:

# pip install psycopg2
Collecting psycopg2
  Using cached psycopg2-2.6.1.tar.gz
Installing collected packages: psycopg2
  Running setup.py install for psycopg2
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-C8JPB7/psycopg2/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-UIFRep-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
      ...
    running build_ext
    building 'psycopg2._psycopg' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/psycopg
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.6.1 (dt dec pq3 ext lo64)" -DPG_VERSION_HEX=0x090403 -DHAVE_LO64=1 -I/usr/include/python2.7 -I. -I/usr/include/postgresql -I/usr/include/postgresql/9.4/server -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-2.7/psycopg/psycopgmodule.o -Wdeclaration-after-statement
    In file included from psycopg/psycopgmodule.c:27:0:
    ./psycopg/psycopg.h:30:20: fatal error: Python.h: No such file or directory
     #include <Python.h>
                        ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-C8JPB7/psycopg2/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-UIFRep-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-C8JPB7/psycopg2

Granted, the error you will receive when you install a wheel without the external dependencies installed could also be confusing for people new to *NIX:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py", line 50, in <module>
    from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: libpq.so.5: cannot open shared object file: No such file or directory

But I argue that that’s at least better than the previous situation - plus, package authors can always catch that ImportError and provide an error message with a bit more guidance. But the big point is that you only need to have one package installed, no compilers, and no -dev packages. If you are installing psycopg2 there’s a decent likelihood you have the libpq package (on Debian) installed, but not as much of a likelihood you have libpq-dev installed.

And one minor point, all of this is potentially on a production box. These days, especially with Docker, I don’t even like to install compilers on production boxes if I don’t have to.

Direction

I ultimately declined my own pull request on wheel because the Linux platform detection was flawed. As a result, there’s a new library in my versions of pip and wheel that attempts to detect the current linux distribution in a variety of ways, but preferably via /etc/os-release.

As it turns out, the use of /etc/os-release for platform detection had already been hashed out on Distutils-SIG a while ago, but I didn’t discover that thread until today. Thankfully, it looks like my implementation should go along with the proposal discussed there (my implementation was based on Nick Coghlan’s suggestion in my later thread, so it’s no surprise).

Eventually I’d like to have it do illumos distribution detection as well, but that’s a much lower priority.

Changes

ABI Tags

The first problem was that of Python ABI compatibility with differing Unicode widths. Thankfully, PEP 425 provides the specification for how UCS and other ABI incompatibilities should be tagged in built distributions such as wheel. Unfortunately, the ABI tag was not implemented for Python 2 (see related issues, pypi-metadata-format#25, wheel#101, and wheel#63). However, implementing this tag in the pip and wheel packages was relatively easy.

Platform Tags

If the platform tag doesn’t start with Linux, there’s no changes here. On Linux, I append the Linux distribution and distribution version number to the platform portion of the tag. So where the old tag would’ve been linux_x86_64 on a 64-bit x86 CPU, the new tag will read like linux_x86_64_ubuntu_14_04 or linux_x86_64_debian_8.

Thankfully, the PEP 425 reference implementation was written with tag specificity in mind, so in fact when comparing tags, a list of tags, from most to least specific, is checked for compatibility. In the case of the platform tag:

When installing wheels, the wheel with the most-specific matching tag will be installed first. A list of platform tags might look like:
- linux_x86_64_rhel_6_5
- linux_x86_64_rhel_6
- linux_x86_64
- any
When building wheels, the tag for the distribution’s major version will be preferred, for example, linux_x86_64_rhel_6. It is possible to build a wheel for the specific major/minor version using the --plat-name argument to bdist_wheel if the wheel builder has reason to believe it would not be compatible with older minor releases.

See related issue pypi-metadata-formats#15.

binary-compatibility.cfg

As you may have noticed, it’s now possible to build wheels that are targeted for RHEL, but what if you don’t have a RHEL system on which to build, but you do have a CentOS system. Or what if someone built RHEL wheels and you want to install them on your CentOS system. Of course, because they are ABI compatible, it shouldn’t be a problem to install one’s wheels on the other. As a solution for this, Nick Coghlan proposed a file, binary-compatibility.cfg, in JSON format, to be found in either /etc/python or the root of a currently active virtualenv, which defines what platforms the current platform is compatible with. So for example, to build and install RHEL 7 wheels on my CentOS 7 host, I’d write a binary-compatibility.cfg like so:

{
  "linux_x86_64_centos_7": {
    "build": "linux_x86_64_rhel_7",
    "install": ["linux_x86_64_rhel_7"]
  }
}

Ideally, /etc/python/binary-compatibility.cfg would be maintained by distribution vendors.

“Generic” wheels

As mentioned above, it is possible to override the platform tag using the --plat-name argument to bdist_wheel. Because the linux_x86_64 platform remains in the list of platforms for which a wheel will be searched, a wheel builder can choose to build for that platform if it has no non-standard external dependencies.

Anyone execising this option should take care that:

No non-standard libraries (anything other than libc, libm, etc.) are linked by any dynamic object in the built distribution.
The wheel is built on a suitably old platform that it will run on any Linux release within the last ~5 years. Due to symbol versioning in the GNU C Library (glibc), modules built on a newer glibc cannot be used with an older glibc.

Ideally, both of these checks could be automated, and perhaps the decision could be made to utilize the “generic” tag automatically if both 1 and 2 are true, except that “suitably old” glibc versions would need to be a policy decision and updated on a semi-regular basis.

ABI stability

The platform detection code currently makes a rudimentary effort to determine whether a given Linux distribution and version can be considered to have a stable ABI or not. This information is not used, but may be useful for PyPI to decide whether or not to accept wheels for a given platform tag.

Supporting work

docker-build

I don’t have time to write much about it here, but this work was tested using Galaxy docker-build, which received quite a bit of work over the process itself. docker-build is a set of scripts and YAML that we use to build Galaxy tool dependencies (typically scientific software) in Docker, for packaging in the Galaxy Tool Shed.

I extended docker-build to build wheels for as many distributions as had official images in Docker Hub, plus a few more, for both the UCS-2, and UCS-4 variants of Python 2.6 and 2.7. Note to self, I need to push updated *-wheel images to Docker Hub.

Conclusion

I hope this is accepted upstream, although I haven’t seemed to attract much attention with my latest posts to Distutils-SIG. I may try opening PRs shortly and see if that spawns any discussion.

The modifications to wheel are in my wheel repository on Bitbucket. The modifications to pip are in the linux-wheels branch of my pip repository on GitHub.

You can see the wheels that I’ve built using docker-build on our wheel index.