LWN.net Weekly Edition for July 23, 2026

Welcome to the LWN.net Weekly Edition for July 23, 2026

This edition contains the following feature content:

• Debating the role of large language models in the kernel community: the kernel LLM discussion has been more nuanced than many think.
• Save and restore may be coming to GNOME: the pieces are coming together to bring a much-missed X11 feature to GNOME on Wayland.
• Fedora grapples with change: a look at Fedora's ongoing discussions about managing distribution and project changes.
• More coverage from the 2026 Linux Storage, Filesystem, Memory Management, and BPF Summit:
• Attaching programs to multiple tracepoints: Jiri Olsa discusses the changes necessary to make tracepoint attachment more efficient.
• Securing BPF LSMs against tampering: Christian Brauner leads a discussion of the problems systemd has had in deploying BPF LSMs.
• Merging famfs?: famfs is aimed at shared access to huge memory appliances, but it has not found its way into the kernel, so the discussion centered on how to change that.
• Sched-ext: enqueue() for sub-schedulers and proxy-execution support: progress continues in the sched_ext subsystem.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Debating the role of large language models in the kernel community

Like many development communities, the kernel community has been struggling to determine how large language models (LLMs) will be used in its development process. The news has been dominated recently by a strongly worded missive from Linus Torvalds on the subject, but the discussion has been rather more wide-ranging and nuanced than that. Topics that have been considered recently include the LLM attribution requirement, code-review tools, dependence on proprietary tools, and whether there is a place for concerns about the ethics of LLMs.

Reconsidering Assisted-by

The kernel community's guidance regarding LLM-generated code was added to the kernel, at the end of 2025 for the 7.0 release, after long discussions in the Linux Foundation's Technical Advisory Board (TAB) and on the mailing lists. The coding-assistants.rst file states that, when a code submission has been partially or entirely generated by an LLM, the patch should carry an Assisted-by tag in this format:

    Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]

The purpose of this tag was to document the use of an LLM, and to alert reviewers and maintainers to that fact. It also contains the name of the specific tool used. That information was deemed potentially useful in case a specific LLM turns out to have a tendency toward specific bugs or ends up in copyright-related trouble; it would make it possible for developers to review patches generated by the problematic model and, possibly, modify or remove them.

Since then, this tag has not won over all developers. As of 7.2-rc4, over 1,200 commits carry Assisted-by tags. There is, however, clearly a significant stream of machine-generated patches that do not carry that tag; sometimes that is a result of ignorance of the rules, but other times the origin of the code is, seemingly, being deliberately obscured. As a result, the tag's usefulness as an indicator of LLM involvement is unclear at best. Meanwhile, some developers see placing the names of specific LLMs into the kernel's development history as a form of advertising. All told, many wonder whether the tag adds any value at all.

This discussion came to the fore at the beginning of July when Christian Brauner suggested removing the tag or, at least, removing the name of the specific model used:

I acknowledge that my stance is even more radical: imho we would just stop it with any disclosure requirements completely. It's useless imho. We already see that other than core contributors most people don't care and will just not disclose their usage of AI. I think this is entirely pointless and worse it brings in undefined legal status as well. It's not like recent events of pulling certain models from the face of the earth have made this any less concerning.

But fine, if we want to do this can we please just dumb it down to

    Assisted-by: LLM

Jeff Layton later followed this up with a separate patch removing the attribution requirement entirely. Some developers were clearly in favor of that; networking maintainer Jakub Kicinski let it be known that he simply removes those tags from patches he applies, essentially undermining the previous decision made by the TAB and the community. Others have not taken that step, but expressed little love for the tag.

There are developers, though, who would like to see its use continue. Greg Kroah-Hartman described it as a signal that an LLM was involved in the creation of a patch, and that the result should be reviewed more closely. Lorenzo Stoakes characterized the current policy as being "about empowering maintainers to push back". This contingent would like to see the Assisted-by requirement stay in force, perhaps with better enforcement if possible.

In the end, there appeared to be enough support for keeping the tag to prevent its outright removal, but nobody was willing to defend requiring disclosure of the specific tool used. So the consensus seemed to be to go with Brauner's initial suggestion, perhaps with a more focused changelog. A new revision has not yet appeared, though, and it may well be that a final decision on this policy will not be made until the Maintainers Summit in October.

Dependence and sustainability

While the use of LLMs to write code is clearly on the increase in the community, LLMs are still much more widely used for the review of patches rather than their creation. The Sashiko review tool, in particular, has seen extensive adoption by many subsystems, and maintainers increasingly expect developers to respond to the reviews it emits. But these tools depend heavily on the willingness of corporations to fund their use, which drives a couple of concerns: will that generosity continue in the future, and will the community become so dependent on those tools that a future rugpull could substantially hurt the development process?

Roman Gushchin, the most active developer behind Sashiko, raised this concern in his proposal for a Maintainers Summit discussion:

Several kernel engineers and maintainers have rightfully expressed concerns about relying on infrastructure provided by a single company without clear formal guarantees. It would be great to discuss what a more sustainable model could realistically look like and how we might get there.

I have also brought this issue up in the community more than once. Two decades ago, the community's reliance on BitKeeper led to a major shock when access to that proprietary tool was suddenly withdrawn. Depending on another proprietary tool now risks a similar shock, and this time the community may not be able to depend on Torvalds creating a replacement over a long weekend.

Many of the responses to this question have leaned on the idea that, while LLM-based tools help the community to be more productive, there is no real risk of becoming dependent on them. Sasha Levin, for example, said: "if AI suddenly goes away tomorrow it'll suck for me, but I'll just go back to writing my tools manually like I did before". The earlier part of that email, though, read:

In the past year or two I was able to rewrite most of my ugly scripts, automate so many of the processes I used to do manually, and just improve so many quality of life items thanks to AI. Look even at the CVE process that was created last year: so much of the infrastructure used to drive it was created with AI.

Or consider the words of another long-time kernel developer, Dave Chinner, in an entirely different conversation:

I have learnt how to drive LLMs well enough that I don't need to write code anymore. The LLM functions as my code editor that is capable of extremely fancy DWIM predictive text insertion. For someone who hates the process of typing out code, this has been a revelation.

The authors of words like that may not feel dependent on these tools, but they have nonetheless clearly given LLM-based tools a major role in how they get their work done.

Ted Ts'o said that Sashiko finds problems that human reviewers might not notice, but that it is not a substitute for human review: "I agree that if reviewers stop doing this, we would be in a world of hurt". He also said, in a different discussion, that "one of the things that really excites me about Sashiko is that it reduces my workload". How maintainers will respond to LLM-based reviews in the long term remains to be seen, but it seems almost certain that they will take advantage of that workload reduction and stop looking for the sorts of problems that tools like Sashiko tend to find anyway. It is hard to imagine that the skills needed to do that kind of review would not atrophy over the long term in such a scenario.

If LLM-based review tools are with us for the long term, the loss of those skills might be no more tragic than the loss of low-level assembly skills. They just will not be needed anymore, and humans can focus on more strategic and long-term-maintenance issues. But if those review tools disappear as the pipeline of money behind the LLM industry constricts, development communities could find themselves regretting that skill loss. Compilers cannot be taken away from us; massive, data-center-based LLMs can. Even if the tools do not disappear entirely, they may become unavailable to developers who do not work for large companies that can pay the bills for their use.

There are some developers looking into ways of running tools like Sashiko locally, using open-weight models; Mauro Carvalho Chehab and Takashi Iwai have mentioned such efforts. According to Iwai, the results are not yet the same: "Yes, there are definitely many false-positives, and they don't always follow the recent changes. Also they cover much less changes than Sashiko". But, presumably, these efforts will make progress over time, reducing the level of dependency on the resources supplied by companies.

Email from Sashiko

To what extent should contributors be required to use — or, at least, to interact with — LLM-based tools? The 2026 Media Summit, held in May, discussed the use of Sashiko for media patches. Some of the outcome of that discussion can be seen in this email from Chehab (the maintainer of the media subsystem), who said that a separate email list had been set up to receive Sashiko emails to keep them off the main media mailing list. He also said that there is a desire for individual contributors to be able to opt out of receiving Sashiko reviews of their patches. That led to a rather lengthy discussion.

Gushchin responded that an opt-out capability seemed like a strange request, like asking to be able to opt out of unit tests. Chehab agreed with him, saying that community etiquette called for always copying the author on replies, but that there are members of the community who see things differently. Laurent Pinchart answered that etiquette called for an explicit request before signing somebody up for a new mailing list. When Gushchin asked whether even a "critical issue" found by Sashiko should not be sent to a patch author, Pinchart responded: "I expect maintainers who want to act on sashiko reviews to triage and verify them first before bothering authors".

It is fair to say that the idea of expecting maintainers to review Sashiko's reviews before sending them to contributors was not entirely popular with maintainers. Ts'o refused that duty and said, instead, that he feels no need to accept patches from contributors who refuse to look at Sashiko reviews. Ihor Solodrai said that it is the contributor's job to make a patch good enough to be merged and, in the current environment, that includes addressing bugs found by LLM-based tools. Gushchin pointed out that attackers are using these tools even if contributors reject them. James Bottomley added that "the contributor doesn't get to approve the tools the maintainer uses to assess and apply patches".

Pinchart pushed back, though, saying that, since maintainers are forcing the change toward these tools, it is up to them to use the time the tools are saving to triage the reviews instead. To require dealing with Sashiko, he said, is to "force contributors to constantly justify their value against a machine that is known to produce a non-negligible quantity of nonsense". Having to respond to unchecked reviews from LLMs, he said, is equivalent to maintainers being forced to read LLM-generated patches from contributors who have not checked them. He also referred back to the Media-Summit discussion about sending machine-generated reviews to newer developers:

This came from a consideration on the impact of false positives and other hallucinations on junior developers. They may have the best context of the work, but they are also most at risk of not questioning the validity of the reviews.

A related area of concern, raised by a number of developers (including Gushchin in his proposal for the Maintainers Summit) is that of pre-existing issues. As Sashiko reviews a patch, it also finds bugs that already exist in the code being changed. Surprisingly, it would seem that the current kernel code base still contains no shortage of errors. Nobody seems to think that these bugs should not be reported, but many see them as noise that the developer of the patch being reviewed should not have to work through. It would be nice, certainly, if the developer would fix all of those bugs, but it is generally not considered reasonable to require developers to do that extra work.

As Torvalds noted, the tendency of some maintainers to expect contributors to fix unrelated problems has been a pain point in the past. It doesn't work, and should not be done with bugs found by LLMs either. He had a somewhat eye-opening suggestion that, perhaps, an LLM could create its own fixes for at least some of those bugs and maintain a tree of its own patches. "That would seem to be the obvious 'next step' from just doing patch review. Trivial patches that the AI has high confidence in, and we have high confidence in the group running the AI". There was no immediate clamor of support for that idea. Instead, developers talked mostly about trying to somehow maintain a separate list of pre-existing bugs that need attention when somebody has the time to do so.

Forking and ethics

Returning to the media discussion; Gushchin had responded to the request for a per-developer opt-out capability by describing the request as "a very anti-LLM position in general" and something that would defeat the purpose of helping maintainers. That led to the Torvalds response reported on around the world:

Linux is not one of those anti-AI projects, and if somebody has issues with that, they can do the open-source thing and fork it.

Or just walk away.

AI is a tool, just like other tools we use. And it's clearly a useful one.

Lyude Paul's response to Torvalds is worth reading in its entirety. She points out that many developers need their paychecks and are not in a position either to fork the kernel or to walk away from it. Developers are currently under heavy pressure to use LLM-based tools, to the detriment of the quality of the code produced:

Many, many of our employers are explicitly trying to force these tools on people with the expected result being a magical 10x productivity boost that just isn't compatible with oversight. It is being used in a LOT of workplaces as an excuse to throw any notion of actual review or code quality out the window. While we've been dealing with slop for decades in the kernel already: I think it is very worthwhile to point out that a tool that dramatically lowers the bar required to generate huge amounts of sniff-test passing slop is in fact, not like other tools.

For developers who are working to maintain the quality of the kernel's code, Paul said, guidelines on LLM use within the kernel may be the only tool those developers have to push back safely on their employers.

This pressure from employers has also arguably had the result of making many developers unwilling to argue publicly against their use, leaving Pinchart as one of the few willing to do so. He made his position clear on the topic: "I consider that, today, there's no ethical justification for the use of generative AI in FOSS development". Torvalds was quick to shut down any discussion along those lines: "So keep your ethics where they belong - in your personal life. Don't try to enforce your ethics on others".

This position is worth reflecting upon for a moment. Developers will, given a chance, describe a number of ethical misgivings about these tools. They will talk about the industry's decision to ignore copyright and the possibility that LLM output violates copyrights. They may mention the environmental costs of untold gigawatts of computing power being directed toward these models, or the effects of data-center construction on the communities around them. Developers may express concerns about how these models are controlled by oligarchs, or about the systemic economic risk posed by the huge portion of the economy being pulled into this industry. They might even spare a moment to commiserate with web-site operators who are struggling to contend with scraping by bot networks.

Whether or not one agrees (or cares) about these concerns, there is no doubt that they exist. But, according to Torvalds, that entire discussion is off limits; the only aspect of the problem that can be talked about is "technical reasons" and whether the tools are useful. Clearly, the kernel community is not in a position to fix any of those problems. It is also clear that the community cannot pretend that these tools do not exist. But neither can the community pretend that the ethical issues around LLM use do not exist, at least, not without the risk of alienating the developers who do care about these things.

As anybody who has made it to the end of this overly long article will have long since realized, there are a lot of open questions regarding the use of LLM technology in the kernel community. Many developers and maintainers are clearly quite supportive of that use, and it seems clear that the role of LLMs in kernel development is set to grow. This community has muddled through numerous debates over the years about its direction and, yes, about its ethics. The result has generally been a stronger and larger community going forward. Hopefully, the community will find ways to reach that sort of outcome this time as well.

Comments (132 posted)

Save and restore may be coming to GNOME

One of the features that users often miss when moving from X11 to Wayland is the ability to save and restore the position of windows between sessions. At GUADEC 2026, held in A Coruña, Spain, Adrian Vovk provided an overview of work that has gone into providing a platform-wide save and restore framework for GNOME. After two failed attempts at landing an API, he believes that the third try will be the one to succeed—though not in time for the upcoming GNOME 51 release due in October.

I did not attend GUADEC in person this year, but I watched the video of Vovk's talk after it was streamed. The individual talk videos have not been posted yet, but his talk is available at 5:40 in the full video from day two of the conference. The slides from the session have not yet been posted.

The idea behind the save and restore feature, Vovk said, is simple. A person is using their system and has applications open while working on something, then they need to log out of the desktop or reboot for some reason. When they log back into their desktop, they would like to restore the previous session: that is, application windows resume their previous positions and sizes, re-open the same documents, and so forth. Currently this is not possible in GNOME, at least not without additional extensions.

There are GNOME extensions that try to emulate this behavior, such as "Restore Geometry" and the creatively titled "Another Window Session Manager". However, the functionality is arguably something that should be available in GNOME itself.

Benefits

Save and restore is a commonly requested feature for GNOME, Vovk said. Its absence is also something that people often cite as a deficiency in Wayland when comparing it to X11. It has practical benefits, he noted, in recovering from application and system crashes.

He said one of the benefits he found most interesting about save and restore was that it could encourage users to update more often. Updates often require rebooting the system, and users of immutable systems must reboot to apply updates: "However, rebooting the system is quite disruptive. You lose everything you were working on." He admitted that he would get his workspace set up as he liked it and avoid applying updates for one or two weeks so he would not have to start over again.

The groundwork for save and restore could also be used for "other, fancier features". For instance, mobile phone operating systems save the state of an application and shut it down to save power or RAM, then restore it when the user switches back to the application. It might be possible to do something similar on the GNOME desktop. Another way the framework could be used is to implement something like Apple's Handoff feature that allows users to move an application's state on one device to another and continue working on the target device.

Vovk emphasized that he was not promising those features would be implemented or even worked on, but the save and restore framework would set the groundwork for such things.

History

With X11, applications would register with the session manager using the X Session Management Protocol (XSMP), which provided a uniform mechanism to allow sessions to be saved and restored. The applications were responsible for communicating their state and for restoring themselves. "Theoretically we could have used this with Wayland, but it has some problems", Vovk said. He noted that X11 session save and restore had been broken in GNOME for quite a few years; then the code for it was dropped entirely when X11 session support code was removed from GNOME in GNOME 50.

The big problem, he said, is that XSMP was all command-line driven, which meant that an application would provide a command for the session manager to run at the next login, usually with a file that had the application's state saved to it in some fashion. "Of course, that means that none of this works with a sandbox. I mean, we don't want sandboxed apps to be able to tell the operating system to run some commands at login."

To ensure that save and restore works with sandboxing, Vovk's replacement for XSMP would be an XDG desktop portal. This does not diverge too far from the way that XSMP worked, he said. Applications still register with the GNOME session manager, and it still keeps track of what to restart on the next login. Applications are relaunched by using the desktop entry rather than passing a command line for the manager to run. When an application is started again, it is given a "restore reason" which tells it what to do:

So it'll tell the app, "Hey, this is just a normal launch. You should restore maybe your window position, but not the contents inside of the window." Or it can say, "You're doing a full session restore; restore as much as you can", or it could say, "You just crashed, so maybe restore some things, but be careful not to get yourself into a crash loop". And I'm glossing over lots of complexity.

With XSMP on X11, the application was responsible for saving everything itself, including the window-management state. It would ask the X server for its position, size, "and all the window-management flags it felt like asking about. Then it would write them down and later restore them."

However, on Wayland, applications cannot position themselves for lots of reasons, Vovk said, though he did not elaborate on those reasons. Instead, Wayland offers the xdg_session_management_v1 protocol. The application does not save its own state, "it asks the compositor, 'hey, can you save my state and give me a token?' which is just a big random number, essentially". The compositor, such as GNOME's mutter, saves everything. Later, the application can provide the token to the compositor, which will restore "whatever makes sense to restore".

Vovk said that it was better than what GNOME had with X11. First, it was more consistent: what is saved and restored is centralized with the compositor, which can execute policy that makes sense for the desktop. For example, he explained that when an application launches, it might restore its position, but users may not want the application to restore the workspace it had been on previously as well. "It would be really weird if you were on the third workspace and you launched the app, and it opens on the first workspace because that's where it was last time".

With the "old X11 way of doing things", an application would need to be aware of workspaces, which it might not have been, and handle those edge cases. "And different desktops do things differently, and it becomes this whole mess." With the Wayland protocol, Vovk said, the compositor can decide what makes sense to do when starting an application. "During normal launches on this desktop environment, it doesn't make sense to restore workspaces, so it doesn't do that".

GTK

The hardest part of all this, Vovk said, was GTK. What was missing with XSMP was a "nice, high-level API for apps to interact with session save and restore, and so it wasn't that popularly implemented by apps". If applications are going to support save and restore, then they need an easy way of hooking into this system. The overall approach that he wanted for GTK was for Gtk.Application(), which handles session management, to manage everything. "It talks to the portal, it handles the Wayland protocol, and so on." The implementation would be built around what he called "state snapshots".

The idea was that GTK can decide to collect all of an application's state and save it to be restored at a later date. There were quite a few edge cases to consider, he said. For example, if an application has a saved state and the user double-clicks a file in the Nautilus file manager that is associated with the application, "does it restore first? Does it open a new file to handle the file? It's not really clear." He said that it was similar to the case where an application was already running with a couple of windows open already, and then it was sent an activation request to open a new file.

So my thinking here is that restoration isn't really that different from an app that's been running for a while and the user's already done some things in there. So what GTK will do is when you launch the app, it will restore the app first. And that will bring the app up to the state where conceptually it's the same state as if the user has been using the app for a while, has a couple windows open, and so on.

And then we give the app the activation request. And that means that all the edge cases of save, restore, lifecycle kind of go away because we pretend that that case is the same as the case of an app already running.

GTK has a periodic auto-save feature, he said, for crash resilience. Every 15 seconds, by default, GTK will take a snapshot of the application's state. If there is a crash "you lose at most 15 seconds of what you were doing in your last-focused app". The goal is for GTK to hide all the complexity, he said, of the Wayland protocol and edge cases. He described a scenario where a user closes an application and then restarts it before it completely exits. "Maybe the app is being held running by a search provider, or it's running some task in the background, whatever." GTK needs to restore the application window, but not the parts of the application that were "still alive".

First API

The first API that made it into GTK was "quite simple" and completely synchronous, he said. "We just emit the signal, and when all the signal handlers return, we assume we're done. We assume all the state has been collected." It was pretty close to landing in the final releases GNOME 50 and GTK 4.22, which were announced on March 18, 2026. "We almost released this." The first API had a few drawbacks, however, which held it back.

Some of the drawbacks were that the implementation required "a lot of repetitive kind of code", and that it didn't take into account an application's full hierarchy of windows. The first API was not well-suited for the kind of navigational hierarchy, such as subpages with their own states, that an application would have. "There was no helper for building up a tree of state that you would want to then kind of pass down your navigation hierarchy."

It also lacked bulk data support. He cited the example of GNOME's text editor: "You may have all the window state you want to restore, but then you also have this unfinished text document in there that may be many megabytes in size." It was undesirable to copy that text document into the save and restore system, "because then we're copying the data into memory and we're then writing it all to disk, and it's so much unnecessary work". It would be better to save the data to a file "off to the side" and keep the state snapshot small so that the entire system is quick.

But the biggest drawback of the first API, he said, was that applications want to do asynchronous things, but the API was fully synchronous. In the GNOME text editor example, saving the text document to disk should be an asynchronous operation: the file could be large, the disk could be slow, "we don't want to hang the app while we serialize everything".

The asynchronous problem is what killed the first API, Vovk said. This issue came up, Vovk said, because he had published the API for developers to test and there were a few applications adapted to use it. "One of the big pieces of feedback was, 'well, I want to do something async here, but I can't.'". He had to tell them "actually, you can't do that". He thought originally that it would be possible to retrofit asynchronous support into the API, but it involved "a big, big pile of race conditions that we couldn't fix". So, shortly before the GTK 4.22 release, "we pulled this API. We just hid it, basically. All the public API pieces got deleted, and the plumbing is still inside GTK, but it got hidden".

Second and third try

After the GNOME 50 release, he hoped that it'd be possible to implement a second API in time for GNOME 51. His second attempt had asynchronous support, managed window hierarchy, and had a solution for bulk data as well; but the way it was implemented, using asynchronous GObject signals, was "overwhelmingly rejected" by the maintainers. "The bit of feedback I got was that async signals is not something people are interested in, which, fair enough."

For the last few weeks, Vovk has been working on a third API proposal that keeps the benefits of the second API with a slightly different way of doing it. There are no more async signals; it uses asynchronous methods or virtual functions (vfuncs). "It's just an interface, and there's no weird machinery."

"So, when is all of this landing?" Vovk asked rhetorically. The Wayland bits are mostly there and the protocol has been released. It is ready for use by toolkits and compositors. The mutter side of save and restore landed in GNOME 51 beta, "however, there are some bugs to iron out when you actually put this together in a full session". The Wayland parts for GTK were released in GNOME 51 alpha but it is not usable by applications yet.

The XDG session save/restore portal is not quite done yet. He said that he had talked to KDE developers and Firefox developers who had agreed with the core concepts of the portal, but the details still need work, such as the specific restore reasons and some of the other language. The work is being tracked in this pull request on GitHub. He is also working on an API for the GNOME Session Manager, and there is a merge request that is waiting for other pieces to land.

The basics of the third API are in a branch awaiting feedback from GTK maintainers, "which I'm sure I will be getting at some point within the next few weeks". He was not sure what the reaction would be, yet. He said he would like to have some applications ported to the API before it's made stable in GTK. "We really need to be aware of what the ergonomics look like before using it, before stabilizing it." Ultimately, he said, it will not make it in time for GNOME 51. He encouraged developers with feedback on the API to contact him.

Questions

One audience member had a question that was difficult to hear, but Vovk thought they wanted to know if there would be an option to turn save and restore off. He said that there would be a user-facing option in GNOME's settings application to turn it off, and noted that applications would have to opt in to supporting the API as well.

Another person wanted to know if it would be possible to implement the save and restore feature at a lower level than GTK. Vovk said that the problem was that it needed to use GtkApplication() because there is involvement with Wayland, and with talking to an XDG portal, "which GTK will do, but GLib isn't doing". The feature also needs to be aware of GTK windows, "because otherwise we get all these edge cases you have to handle yourself" in an application, which he did not think was feasible.

The final question was about the GNOME extensions that currently implement save and restore features; what would the introduction of a proper GNOME feature mean for those extensions? He said that the extension probably would not be necessary long-term, but there would be a transition period while applications were ported to use the feature that it might make sense to continue using the extensions.

Comments (14 posted)

Fedora grapples with change

The Fedora Project is known for, among other things, having a well-defined set of processes for just about everything. It has extensive packaging guidelines that deal with the complexities of creating RPMs to install software, as well as processes for managing the legal questions that arise around shipping software. Fedora also has a well-defined change process for dealing with self-contained technical changes as well as major changes to the distribution, and other issues as they arise. At the moment, though, the project seems to be experiencing a sort of midlife crisis as it re-examines several of its change processes at once to determine if they are still effective.

Evolution

As a vendor-sponsored project, Fedora has always had to walk a difficult path in serving its users, contributor community, and corporate masters; what makes one party the happiest may be a source of angst for another. Red Hat's desire to trial technologies in Fedora does not always spark joy among contributors and users. Volunteer contributors may want to push ideas that are of little interest to Red Hat (at least at the time), or may even conflict with its choices. For example, Fedora's choice of Btrfs as the default filesystem contrasts with Red Hat's decision not to support it in Red Hat Enterprise Linux (RHEL).

Users want easy access to problematic software, such as patent-encumbered games or codecs, but shipping those could create expensive legal headaches for Red Hat. Attempts to make Fedora more user-friendly, say by setting the default for the $EDITOR environment variable to GNU nano, may not please some developers.

Fedora had little in the way of governance or policy when the project launched in 2003 as a kind of replacement for Red Hat Linux. Early on, Red Hat entertained the idea of creating a "Fedora Foundation" that would give the project some independence, and then pulled back from that in 2006. Some of the mechanisms that were put in place in anticipation of the foundation, such as a Fedora Board and Fedora Extras Steering Committee, evolved into the Fedora Council and Fedora Engineering Steering Committee (FESCo) over time.

There have been a number of times when there was friction between corporate and community priorities; but the project has more or less made it work over the years. It has done this by discussing the problems, finding some kind of consensus, documenting the policies that are hammered out along the way, and then using them for new decisions. See, for example, Tom Callaway's overview of how Fedora's legal policies were developed. Over the years, Fedora's governance and policies have become something of a model for other open-source projects.

The project's governance has had one constant: Red Hat has the final say. As former Fedora Project Leader (FPL) Max Spevack noted when Red Hat dumped the idea of an independent foundation:

Red Hat *must* maintain a certain amount of control over Fedora decisions, because Red Hat's business model *depends* upon Fedora. Red Hat contributes millions of dollars in staff and resources to the success of Fedora, and Red Hat also accepts all of the legal risk for Fedora. Therefore, Red Hat will sometimes need to make tough decisions about Fedora. We won't do it often, and when we do, we will discuss the rationale behind such decisions as openly as we can.

Just because Red Hat has power over Fedora does not mean that the company wants to use it, he said. Nor did it want to make all the important decisions about Fedora: effective community-driven decision making would be a direct measure of Fedora's success. "We aim to set the standard for open source innovation. A truly open Fedora Project is what makes that possible."

Sandbox

In the past year or so, though, the project's processes that have worked until now seem to be coming into question more and more often: usually when they seem to butt up against Red Hat's priorities. For example, last year's deliberation on Fedora's AI-assisted contribution policy that showed a disconnect between Red Hat's desire to experiment with AI in Fedora and contributors who wanted Fedora's policy to be much less friendly to AI.

In March, current FPL Jef Spaleta proposed a technology-innovation-lifecycle process he dubbed the "Fedora Sandbox" for "experimental features, components, output, process, or services".

The driver for Spaleta's sandbox idea seemed to be related to frustrations that Red Hat leadership had with pushing its experiments into Fedora; the project's existing processes, which involve compliance with Fedora's strict packaging and license policies as well as a need to gain community buy-in, could slow or stymie acceptance of things Red Hat was going to do for RHEL anyway.

Having to maintain those projects outside of Fedora, which ultimately serves as the foundation for a RHEL release, is a likely source of frustration for folks working on both—not to mention their managers and Red Hat leadership who (not unreasonably) just want features to land in RHEL on schedule to make customers happy.

Spaleta's sandbox process would have allowed experiments to be conducted in Fedora even if they broke "non legally binding" policies as long as they had a "reasonable path forward towards resolution" before being fully integrated into the project or distribution. He proposed that the sandbox would be in addition to the existing change processes and community initiatives that can be used to set longer-term goals for Fedora, such as the completed initiative to create the Fedora IoT project, or the current Git Forge initiative to replace the Pagure collaboration platform with the Forgejo-based Fedora Forge service. The sandbox proposal has not been accepted (or rejected) yet.

Initiative friction

Red Hat developer Gordon Messmer proposed an AI developer desktop initiative, on March 31, with an aim to "build a thriving community around AI technologies" within Fedora. That proposal met with some opposition from the community, with objections ranging from a dislike of AI to complaints about potentially changing Fedora's policy for out-of-tree kernel modules. Ultimately the proposal was discussed by the Fedora Council, initially approved, and then blocked at the last minute when council member Justin Wheeler changed his vote on May 8. Council member Miro Hrončok, likewise, also changed his vote on May 13 saying that "the Fedora community is not supportive of this initiative as is".

Another Red Hat idea, an automated approach to building an operating system called Project Hummingbird, was raised on the Fedora development list in April. It was greeted with some interest, as well as some confusion about how it might differ from other Fedora variants, but it didn't seem to face much opposition. It was never formally proposed, though McCarty said he planned to do so through Spaleta's sandbox initiative.

Instead, Red Hat bypassed public processes entirely. It asked the council in private for approval to use the Fedora trademark so that it could announce the project at Red Hat Summit on May 12. Fedora contributors outside of Red Hat were surprised and confused by the announcement. Michael Gruber, for example, wondered whether Hummingbird was legitimately a Fedora project or not. "There might be even some good ideas in there, but given how this started and how it is communicated I can put zero trust in this."

Red Hat employee and Fedora contributor Adam Williamson said it would have been ideal for Red Hat to make the request more openly, but he was happy that the company was trying to do the Hummingbird project within Fedora rather than doing an end run around the project. The more Red Hat has to do outside Fedora, he reasoned, the greater the risk that the company will question its funding of the project.

From inside Red Hat, especially if you don't work on Fedora, it is possible/tempting to look at Fedora as a source of strife. It's got all these people in it, with opinions, who aren't on the payroll! You can't tell them what to do or think or complain to their manager! They have eternal arguments about everything! You have to write a wiki page and convince some person you've never heard of that your idea is good! Who needs this?

So we're kind of constantly fighting a tendency for RH to just spin stuff up in channels it 100% controls, which tends to seem easiest at first then turn out to be a mess after a few years.

Christopher Klooz, however, worried that the funding justification meant that Fedora could be "step by step transformed into a corporate unit of RH". He added that it was already unclear "when the Council acts as agent of RH and when as agent of the community".

Williamson replied that he understood the point, but said that this was a "an ever-present tension" that Fedora has had to manage nearly as long as it's been around:

The way Fedora can "compete" is not by turning into the thing some parts of RH might superficially think they want - an upstream project that's open source in license terms but closely-controlled in governance - but by being the thing they need - a loosely-coupled upstream project with passionate and involved people who will make things awkward and uncomfortable at times but usually produce a better end result over the long term, and act as a useful check on RH's perceptions at times.

Red Hat employees who care about Fedora, he said, have to keep selling that vision and making sure that it works. The Hummingbird discussion petered out not long after Williamson's reply, but it appears that Fedora's decision-making bodies have been mulling over how things are done.

Hit pause

On July 1, Aoife Moloney, Fedora's "change wrangler", announced that the Fedora Council was proposing a pause to Fedora's community initiatives process. Existing initiatives would continue as planned, she said, though "the administrative framework around them may evolve".

The AI developer desktop, she said, had shown that the initiatives process had failed as a framework "where new ideas can surface, receive respectful feedback, and gain Council support for work that fits the project's present and/or future". The nature of the failure, however, was unspecified. One can imagine that the various participants in that discussion might well agree that something had not gone well, but what that something was would depend entirely on the observer's point of view.

The council, Moloney said, wanted to work out a new method of setting strategic direction "in an open, transparent way that more intentionally includes the community voice." The council recognized that it needed to be "better at being more open in our discussions and decision making" because much of the work that leads to proposals "happens under the radar before official approval processes kick in". At the moment the existing "approval pipeline" for initiatives involves the council performing trademark review, and then FESCo reviewing change proposals. That works well, she said, but misses "early and inclusive discussion for everyone across the project". Therefore, the council would be looking at the sandbox proposal closely as an alternative to initiatives or as a complement to some other process that it might develop.

The announcement phrased the pause as a proposal, but Moloney also closed the discussion on the AI developer desktop proposal saying that the council was now unable to consider it "as a result of halting the Community Initiatives process." The council would return to the question of "whether Community initiatives should be retired or revamped once this discussion has reached some kind of conclusion".

Changes to changes

Shortly before the council had made its announcement, FESCo member "Maxwell G" started a discussion on the Fedora development mailing list to gather feedback on how the changes process could be improved. He said he was not proposing anything specific, but wanted to throw out some ideas and see what other people thought.

For example, he wondered if it was time to move away from using Fedora's wiki for proposals, as well as the Wikitext formatting that goes along with it. The formatting often escapes the wiki and makes its way to other fora; for example, the Btrfs change proposal for Fedora 33 from 2020 has a combination of plain text, MediaWiki formatting, and HTML <span> tags that makes it unpleasant to attempt to decipher in any mail client.

He suggested that changes should be in Markdown or "some format that can easily be converted", and stored as text files in a Git repository rather than on the wiki. The change owners could file proposals as a merge request, which would be reviewed and merged by the change wrangler. He also floated the idea of moving away from the Discourse forum as the primary source of truth for change proposals; he also wanted to stop the practice of having discussions on changes take place both on the forum and Fedora's development list:

Cross-posting to Discourse was proposed as an experiment in fesco [ticket] #2989, but there was never a decision made about whether to stop or continue with the experiment. I find the fractured discussions between devel@ and Discourse hard to follow. I think the Discourse setup makes it easier for discussions to "accelerate" or become toxic or repetitive. I don't think it's any better at handling large threads.

The suggestion to consolidate discussions on the mailing list seemed popular. Michal Schorm agreed that splitting discussions between the mailing list and forum was bad, especially for the change owners who had to follow both. Björn Persson also agreed that the discussions needed to be consolidated, but he observed that it was much worse than just following a forum and mailing list discussion. "When I went through the Change process, the state of the Change was fractured over the wiki, Pagure, Bugzilla and even Gitlab." He counted ten different discussion fora and issue trackers, all in their own silos. "At least Pagure and Bugzilla are good at sending me email when someone writes a comment."

Kevin Fenzi, however, thought that the practice of using the forum for change requests should continue. He acknowledged that there were some problems with that, but it meant Fedora received feedback "from people who are not otherwise involved and sometimes [it is] very useful". He felt that it was important to hear from new voices and to accept feedback "in the place where they are".

Former Fedora program manager Ben Cotton, who had worked closely with the change process during his tenure, thought that the process should be modernized, but suggested that a better approach would be to think about what the process should look like at a high level, then think about how to implement it.

There was a suggestion from Martin Kolman that Fedora should build a simple web application to manage the change process. It would be less daunting for new contributors, he said, and could be designed to validate information before a change was submitted. The idea of another homegrown application to maintain, however, did not win supporters. Daniel P. Berrangé said that Fedora had "a long (and disappointing) track record" of building things and then being unable to support them in the long run. "I can't see it being a good use of resources to build & maintain a custom app for this."

Moloney, who works with changes as part of her job, weighed in on June 30. She said she had agreed with some of the suggestions, such as restricting change discussions to the mailing list, but asked that participants in the discussion ask themselves how much of hands-on involvement they had with the mechanics of the process "before suggesting ways to engineer a brand new way to do it". She also noted that there were many moving parts already, such as the move to Fedora Forge, and thought it might be a good idea to let the dust settle from those "and then see what we're missing".

Red Hat engineering manager Brendan Conoboy also spoke up with his view of Fedora's processes from the outside looking in. Overall, he thought that the change processes work pretty well for the Linux distribution. "Here's what doesn't work: changes that fit poorly inside existing processes." He cited the recent two-factor authentication discussion and the AI developer desktop proposal as examples. "The bar to clear to initiate change in a way every interest feels respected is tough to such a degree that even initiating a conversation is fraught."

Maxwell G pointed out that the AI discussion "hit a lot of pain points", some of which were procedural (such as the council failing to clearly announce its intent to vote on the proposal), but there were also social and political disagreements as well. In any case, he said, that was an initiative rather than a change proposal—and that process was under separate review by the council. Conoboy replied, "we knew the [AI] subject matter was controversial, and it seemed that following a documented process would provide a better framework for constructive feedback". Perhaps it did, he said, but it might have gone better if the idea had been socialized with FESCo first.

Outcome

After the discussion had wound down Maxwell G replied on July 7, with his summary of the conversation. His first takeaway was that he should have had a more detailed problem statement that had more context about what he was looking for from the discussion. He found that there was some support for a Git-based workflow, but the wiki-based workflow also had many supporters; many ideas were put forth, but no clear winners.

What was clear, however, was that the split between the mailing list and forum "creates a frustrating burden for Change Owners and other people trying to follow the discussion". The project had tried the "split-brain approach for development discussion" for three years, and it was not working. He said he would focus on solving that pain point first, and then would come back with ideas for a different wiki-based process at a later date.

Given that it is vacation season in much of the world, there is a good chance that the various efforts to modify Fedora's processes will see little movement in the next month or two. It seems likely, though, that there will be some shakeups to the project's processes once Fedora contributors return from holiday and pick up where these conversations have left off.

Comments (11 posted)

Attaching programs to multiple tracepoints

Tracepoints in the kernel are useful for a variety of purposes: debugging, active monitoring, and performance measurements, among other things. Previously, any given BPF program could only be attached to a single tracepoint. Jiri Olsa has been working to change that, and led a discussion about his progress at the 2026 Linux Storage, Filesystem, Memory-Management, and BPF Summit. That work has since been merged, and can be expected as part of the 7.2 kernel.

Other similar facilities in the kernel support multiple attachments; kprobes in particular support it, and can do most of the same things as tracepoints. The difference is one of performance, Olsa said. Tracepoints are slower to set up, but measurably faster to execute. His slides included measurements of how many millions of executions per second each method could manage. Depending on exactly how the tracepoint and kprobe are attached, tracepoints can be about twice as fast. If tracepoints could be made to support multiple attachment without a regression in performance, they would be an attractive option for monitoring performance-sensitive operations.

Olsa then walked through a bit of the history of how tracepoints have been implemented. They use run-time code patching to redirect execution from a traced function to a generated trampoline that executes the attached hook before returning to the traced function. Originally, tracepoints were implemented using the old ftrace API, which has one-to-one correspondences between trampolines, functions, and ftrace objects. That is obviously not conducive to supporting multiple attachment. Menglong Dong tried a different approach that used a single, global trampoline for all tracepoints. That caused a performance regression because the trampoline needed to identify which function had been called, and the overhead of making that determination was too high.

The new approach is to use a newer ftrace API that supports using a single ftrace object to configure multiple functions. Each function is still assigned its own trampoline, but the management of the trampolines can be handled as a group. Trampoline generation is fast, so regenerating several trampolines at once does not have a huge performance impact.

One complication during development of the new multiple-attachment code was the handling of locking. Originally, each trampoline had its own lock that the code had to take in order to modify it. When many trampolines needed to be modified simultaneously, that could result in the code taking a large number of locks, bumping up against lockdep's 48-lock limit and causing problems for debugging. The solution was for Andrii Nakryiko to remove the locks from the trampolines and replace them with a pool of 32 locks that are shared depending on trampoline address; in practice, this allows almost as much concurrency when accessing trampolines, since the locks are only held briefly.

Another problem — which was identified by the Sashiko patch-review system — is the interaction between tracepoints and live kernel patches. The latter also use the ftrace API in order to inform the ftrace subsystem when a live kernel patch has replaced a traced function. When such a function has a tracepoint attached to it, ftrace will notify the tracepoint code that the trampoline needs to be regenerated. Doing so requires taking the trampoline's assigned lock, but the thread might already hold locks (presumably in the ftrace subsystem, although Olsa did not specify) that are acquired after the trampoline's lock by other code, leading to a lock inversion — a deadlock. To remedy that, the code tries to acquire the lock in a loop, sleeping between iterations to temporarily release any mutexes that the thread holds.

That isn't the bug, though: the problem is what happens when a live patch is removed. In that circumstance, the trampoline must be changed back, but the state of the locks is such that the code cannot attempt to acquire the trampoline lock in a loop. Therefore, if the trampoline lock is held when a live patch is removed, the trampoline may not be changed to refer to to the previous location of the traced function. Steven Rostedt suggested that this case should just return EAGAIN so that the live-patch-removal code knows to release locks and try again. Another developer thought that it was okay to ignore the problem, explaining that it was only a performance problem, not a correctness problem, because the only downside of not regenerating the trampoline in this case is an extra indirection from the location of the removed patch back to the original location of the function.

The actual API to attach a BPF program to multiple tracepoints is relatively straightforward, taking a pointer to a program, a pattern specifying which tracepoints to attach to, and an optional set of cookies to let the BPF program know which tracepoint it has been called from.

    struct bpf_link *
    bpf_program__attach_tracing_multi(
        const struct bpf_program *prog,
        const char *pattern,
        const struct bpf_tracing_multi_opts *opts);
                                     
    struct bpf_tracing_multi_opts {
        size_t sz;
        __u32 *ids;
        __u64 *cookies;
        size_t cnt;
        size_t :0;
    };

The API supports tracing function entries, exits, or both. It doesn't support altering the return values of functions or attaching to Linux security-module hooks. Nakryiko asked whether that was a limitation of the design, or if support for the other kinds of tracepoints could be added later. Olsa affirmed that the other kinds were possible to support, he just wanted to focus on the simplest cases first. He has not yet shared any patches to address the more complicated cases.

Another limitation of this interface is what the BPF verifier can ensure, and therefore what is safe for attached BPF programs to do, Olsa explained. When attaching to a single tracepoint, the verifier can check the BPF program's access to that function's arguments, and therefore let the BPF program safely dereference any pointers to kernel objects that are passed to the function. When attaching to multiple tracepoints, the traced functions could have different signatures, so the verifier can't check the BPF program's accesses in a single pass. Therefore, a BPF program attached to multiple tracepoints is allowed to load the function arguments of the traced functions, but not to dereference any pointers it finds there. That is a limitation that will be difficult to lift without causing performance problems for the new API that obviate much of its benefit.

One person asked how many functions Olsa expected a program to attach to at once, and whether this would cause performance problems by sending multiple inter-processor interrupts (IPIs) as the trampolines are created and attached. Olsa thought that most users of the new API would probably want to attach to hundreds of tracepoints at most — such as hooking into every system call implementation. On the other hand, that should not require hundreds of IPIs; the ftrace API uses some x86-specific tricks to avoid issuing multiple IPIs. That doesn't work on Arm, but support could be added relatively easily.

Rostedt confirmed that he had written the code to handle that on x86, and that adding support for Arm would only require writing a slightly different kind of trampoline. Nakryiko asked whether there was a plan to do that work, to which Rostedt replied: "If someone does it, I won't NAK it."

When pressed for details, Rostedt explained that Arm doesn't have a single instruction that pushes the current instruction pointer to the stack and then jumps directly to an address the way that x86's call does. The equivalent instruction on Arm, bl ("branch-and-link"), leaves the previous instruction pointer in a register. Therefore, patching a trampoline into a function requires altering more than one instruction, which makes an IPI necessary to ensure the CPU doesn't see an invalid intermediate state.

If someone would just implement a trampoline that could be patched in using a single, atomic write in a way that respects Arm's rules for patching executable code, the same IPI-avoiding trick as x86 could be employed, and attaching to multiple tracepoints would become faster, Olsa said. By that point, however, the session had reached the end of its scheduled time, and discussion ended. At the time of writing, nobody has contributed such an Arm trampoline, although Olsa's patch set was merged on June 7.

Comments (none posted)

Securing BPF LSMs against tampering

Since 2020, BPF programs have been able to act as Linux security modules (LSMs). Several projects, including systemd, have been working to use that capability to provide more security to users. Christian Brauner spoke at the 2026 Linux Storage, Filesystem, Memory-Management, and BPF Summit about some of the limitations of using BPF in this way, and the changes he would like to see for systemd's use. In particular, he would like a way to make sure that BPF programs cannot be removed or have their private data tampered with.

When a BPF program is attached to an LSM hook or other kernel resource, it is reference-counted: if the user-space program that submitted the BPF program to the kernel closes the file descriptor for the BPF program (for example, because it crashed), the BPF program is cleaned up automatically. This design keeps things tidy and prevents BPF programs from being leaked, but it exposes a point of vulnerability for BPF LSMs. In addition, it is often possible to interfere with a BPF program's execution by altering the data it has stored in BPF maps in unexpected ways. BPF maps are often used for communication with user-space programs, but they can also be used to store auxiliary data for the BPF program itself, and the BPF program can rely on the correctness of that data. Brauner would like to see both of these behaviors changed, so that BPF security programs, once installed, cannot be trivially circumvented.

The code he has been working on ensures that the computer only executes files that have valid dm-verity signatures, similar to the kernel's integrity measurement architecture (IMA); the security benefit is that it prevents any attacks that rely on overwriting executable files or executing programs from unexpected locations. Only programs that come from a trusted, unmodified disk are able to run. Compared to native IMA, performing the check in BPF allows for greater customization and integration with systemd's existing configuration. That code is working, and the plan is to integrate it into systemd.

The part that Brauner has been having trouble with is ensuring that the BPF components themselves aren't tampered with. "Maybe I'm too dumb, but I found no clean way to safeguard a BPF LSM program from all modifications." He eventually settled on a workaround: using BPF to hook into all of the parts of the kernel that could modify the loaded BPF programs, and disallowing any modification that does not come from the init process, which is trusted to be able to manage things.

There are features that are intended to help with this, he added, but in practice they don't suffice. Signed BPF programs, which were discussed at the summit in 2025, are only protected during loading; they can still be unloaded in the same way as any other program. Exclusive maps, introduced as part of the same effort as signed BPF programs, can help with the problem of altering data stored in BPF maps by limiting access to the maps to trusted programs. They aren't a perfect solution, however, because there is a limit of one exclusive map per BPF program, Brauner said. His program needs something like seven different maps that should all be protected.

Andrii Nakryiko asked Brauner to expand on what about his workaround felt insufficient, and what he would like to see changed. The problem is that he needs to add custom hooks to many places in the kernel, Brauner said. If the BPF maintainers add another feature, he has to know about it and update things, just to keep his BPF programs secure. For example, the fact that the BPF program's skeleton remains mapped in the init process means that he needs to prevent anyone from modifying it. That means blocking features like ptrace(); it's hard to be sure that he has really covered everything that could be affected. It would be much simpler, he said, to be able to mark a BPF program as immutable and then not need to worry about any of that.

Nakryiko has also created self-protecting BPF programs. After loading the program, he records the file descriptors for all of the resources involved, and then attaches programs to all of the LSM BPF hooks for operations involving those file descriptors. That solution works well for him, even if it does add some complexity to the program, and he doesn't really see the need for anything more. Brauner asked why, if Nakryiko didn't see the utility in a simpler solution, he had helped to add exclusive maps to the kernel. Alexei Starovoitov explained that exclusive maps are useful for reasons other than security, and said that he thought the feature already supported multiple maps per program.

One audience member wasn't satisfied with Nakryiko's suggested approach, noting that every BPF LSM will want to do something similar, and that therefore it makes more sense to implement support for immutable BPF programs in the kernel. Liam Wiseheart pointed out that there are a huge number of methods by which someone could interfere with a loaded BPF program — loading a kernel module, writing to raw memory, etc. Ultimately, he wasn't sure that the kind of complete protection that Brauner was asking for was possible, even from within the kernel.

Nakryiko brainstormed a few possible approaches, none of which were fully acceptable to Brauner. Most of Nakryiko's suggestions relied on limiting access to the file descriptors for the BPF program and its resources. Unfortunately, those approaches were complicated by the fact that systemd supports re-execution, Lennart Poettering said — the init process can be restarted, passing over its open file descriptors. That is hard to work around. Brauner eventually accepted that a solution based around protecting the file descriptors could be made to work, but still thought it would be overly complex.

Wiseheart asked why it wasn't possible to just pin the relevant file descriptors. Brauner replied that he thought that had been tried, and it caused extra complexity related to protecting the file descriptor against being reopened or unmounted. Wiseheart continued to express skepticism that protecting a BPF program in this way was even possible; even if only systemd were allowed to communicate with the BPF program, there are still many ways to interfere with a running process.

Nakryiko eventually suggested just incrementing the BPF program's reference count, such that even closing its file descriptor wouldn't reduce its reference count to zero, and therefore the program would always stay loaded. That doesn't prevent user space from interfering with the program's maps, but it's a step in the right direction. Wiseheart suggested disassociating the maps from user space and disallowing their IDs as arguments to the bpf() system call. Brauner agreed that his use case didn't require access to maps from user space after initialization, so isolating a BPF program from user space entirely was acceptable.

At that point, the design to go forward with was, if not settled, at least sufficiently argued-over for the time being. Wiseheart moved on to questioning whether the policy that Brauner was trying to implement with BPF (only running binaries with valid dm-verity signatures) was even possible. He pointed out that uprobes can be used to modify the control-flow of user-space code, which would allow for the use of return-oriented programming or similar techniques to run essentially arbitrary code. Also, interpreters for scripting languages run programs in a way that BPF can't easily differentiate from normal file access.

Brauner agreed that scripts were a problem, and thought that interpreters would need to be taught to perform a check similar to what the kernel does. With that big potential loophole handled, several related policies on executable programs should become possible to enforce. It isn't yet clear which solution the BPF developers will implement, if any, but with the number of people interested in making truly secure BPF LSMs possible, it seems likely that a solution will be forthcoming at some point in the future.

Comments (2 posted)

Merging famfs?

The famfs filesystem, which is meant to provide shared access to huge memory-resident files on CXL and other devices, returned to the Linux Storage, Filesystem, Memory Management, and BPF Summit (LSFMM+BPF) in 2026. It was first discussed at LSFMM+BPF 2024 and a new implementation was described at the 2025 gathering, but it still has not made its way into the kernel; LWN looked at a discussion about merging famfs back in April 2026.

John Groves began the session by noting that he is the creator and maintainer of famfs, which is the "fabric-attached memory filesystem". He works for Micron, which makes devices with large pools of memory that are accessible to multiple servers; those products are coming soon, though early access to them is available to some. There are some "interesting management challenges" for such devices and famfs is meant to allow Linux users to handle them. Since last year's summit, he has become the co-chair of the software and systems working group for the CXL Consortium; "they had the bad judgment to elect me to that", he said with a grin.

He had two different ways of looking at the development of famfs so far. It has been "making things worse for VFS and FUSE since 2023", but the filesystem community has been "making perfect the enemy of good" over that span as well. He hoped his session could help "find a way to stop doing that" so that famfs can make its way into the kernel.

Introduction

As part of his introduction to famfs, Groves stressed that "famfs cannot be used as a general-purpose filesystem". An application cannot simply open a file on famfs and start writing to it; it is, instead, for creating non-sparse files in non-sparse, shareable memory. Famfs is not read-only, but is normally fairly static; data is moved into it and then accessed for analytics or other processing.

Famfs provides a filesystem interface to shareable DAX memory. Access is via regions created with mmap(), while read and write are generally handled by memcpy().

Metadata is managed so that multiple hosts can mount a famfs filesystem; there is a single master node that puts the files into the filesystem to make them available to the client nodes. "File maps" describe the mapping of the file to offsets and ranges on multiple DAX devices. Interleaving data across devices is critical for performance; that's how memory is laid out for system RAM and it needs to be that way for shared memory devices as well. He said that the interleaving mechanism is not CXL-specific at all and could be used for other shared-memory devices that provide a DAX interface.

Ted Ts'o asked whether the master node was the only node that could create or write files. Groves said that for the foreseeable future, only the master can create files, which are fully pre-allocated at creation time. It is usually only the master that writes to the files, as well, but clients can write "and you're responsible for whether you're doing something that makes sense". CXL "makes cache coherency worse", he said to chuckles around the room.

There is no device mapper for memory, Groves said; there are only page tables, translation lookaside buffers (TLBs), and fault handlers. A filesystem can bridge those worlds. But, famfs is not storage, just volatile memory—there is no backing store. Typically, the data of interest is collected into a file on a regular filesystem (XFS, say) by the master node, then copied to a famfs file. Each node in the cluster would then mount the famfs filesystem and run its workload on the data in the file.

There are two separate implementations of famfs, the original standalone version and the more recent FUSE-based filesystem. He has taken the approach of supporting both implementations, which are differentiated by a mount-time option. The FUSE version came about after a discussion at the 2024 summit and it took him most of a year to get that working.

There are 100TB appliances available now, though the memory for them may actually not be available until 2030, he said to knowing laughter. But there is currently no abstraction available to use them in the kernel. The ability to place enormous datasets into memory provides a means to solve problems that are currently impractical. Applications already know how to use files; with famfs, they do not need to be rewritten to take advantage of these new devices.

Amir Goldstein reminded Groves that he should save some time "for the contention point", though Groves joked that he brought his beer so that he could not be deprived of it by a lengthy discussion as was postulated in the April LWN article. Eventually, Christian Brauner asked: "what do you want from us?" Essentially, the answer is "merge famfs", but it took Groves a bit of time to get there.

The dilemma for the FUSE-based implementation is that famfs requires ABI changes to FUSE in order to handle its file maps. But the FUSE-based famfs does not perform as well as the standalone version. He has "been slinging FUSE patches for a year" at this point without much of a response from the FUSE community. Recently, though, there have been objections to the changes needed to the FUSE ABI.

BPF?

As part of the discussion about those ABI changes, it was suggested that BPF be used to calculate the extents based on the interleave information. That would require running a BPF program as a fault handler, which is a clever idea, Groves said, but famfs is performance-critical, so it is not a good candidate to test whether the idea is viable. He feels like he keeps hearing that if he changes X, the FUSE community might accept it, but it is clear that he is growing weary of the frequently shifting ground.

Goldstein summarized the two FUSE ABI changes that famfs needs, one to determine which DAX devices are present and another to provide the file map to the FUSE server so that it can handle the faults correctly. He asked Miklos Szeredi, who is the FUSE maintainer, if those were acceptable. Szeredi said that he was not opposed to merging famfs as it stands. As he said in the discussion of the FUSE famfs version 10 patch set, he would prefer to have an alternative to a famfs-specific interface. At the time, famfs was flying under his radar to a certain extent, so he had not looked in detail at what was being proposed for it; the BPF idea was raised and he is curious about how it would work.

Christoph Hellwig objected to the whole idea of bringing BPF into the picture. He said that the problem with famfs really does not have much to do with the filesystem at all. It is, instead, applying a mapping layer in FUSE that is separate from the one that other filesystems are using: iomap. Darrick Wong has been working on an implementation of iomap for a FUSE-based ext4 that takes the right approach. "We want a user-space FUSE server to do a mapping of a logical offset into a physical layout either on a disk or on DAX and it totally makes sense to include striping." He strongly suggested building a "proper high-level infrastructure" and not hiding this kind of work in filesystem-specific code.

Groves said that he and Wong had spoken months earlier about combining their work, but they came to the conclusion that there are famfs-specific repeating patterns that are used for interleaving the data, which is not something that other filesystems need to do.

An attendee asked about the performance difference for FUSE-based famfs versus standalone. Groves said that he has measured some things, but knows that FUSE is not noted for its performance. Goldstein pointed out that transferring the file map is not particularly performance-sensitive, which Groves acknowledged.

Groves had attempted to run a proof of concept of the BPF idea, which Gregory Price had put together with an LLM's "assistance", but was unable to get it to work. Price agreed that the build environment used by the LLM was non-standard and that mostly what he took away from the experiment was that BPF did not make sense to use for FUSE.

Goldstein said that if there is a way to describe the file maps that Szeredi is willing to add to the FUSE protocol, how it gets implemented right now does not really matter. It could use Groves's code now and iomap later, for example. This is why it is important to generalize new features, Hellwig said; Wong's current ext4 code does not do striping, but other kernel filesystems do (e.g. Btrfs), so that feature will be needed if the FUSE iomap code is to be used more widely.

Hellwig suggested that a combined effort from Wong and Groves that added useful infrastructure for more than just the famfs use case might find a much easier path into the kernel. It sends a different message that shows other developers that the feature affects more than just famfs. Szeredi said that Wong and Groves had already tried to find common ground, but failed.

At that point, Wong chimed in over the remote link. He said that the famfs pattern-based extents could result in huge numbers of mappings that would need to be stored by the kernel in an iomap-based implementation. A file of 100TB with, say, 2MB stripes, results in a huge list of extents that would need to be stored in kernel memory. To him, it did not make sense to do that when famfs can simply calculate that information based on the pattern it is using. "It seems kind of silly to me to go upload billions of mappings into the kernel, just because you don't want to have a little bit of executable code." That was what led to the "crazy BPF thing", which he said "was horrible".

Hellwig said that the formulas for striping patterns are well-known and long-established; there are three parameters that are used to describe them and adding custom executable code for them (e.g. BPF) should not be needed. Groves agreed and said that those parameters are basically what he had used for the FUSE file-maps protocol message. Goldstein said that as long as others could use the file-maps facility for their own purposes, it should be merged into FUSE; the protocol details of the message fields and such will still need to be worked out, however.

Wong suggested that he could take Groves's file-map patches and rename them to "FUSE map" or something generic to get them upstream with his FUSE iomap work. Goldstein agreed that something like that should be possible. Groves jokingly wondered if that meant he would need to wait until 2028 or so, but Brauner pointed out that someone had just offered to do much of the work for him—to more laughter. Goldstein said that it may seem like Wong's patches complicate what Groves is trying to do, but there is a fair amount of overlap and a lot of code that Groves can use for famfs. In the end, progress was made, Goldstein said and Groves agreed, so a path forward for famfs has hopefully been found at this point.

Comments (none posted)

Sched-ext: enqueue() for sub-schedulers and proxy-execution support

The extensible scheduler class (sched_ext) allows the installation of custom CPU schedulers as a set of BPF programs. While sched_ext, in its current form, has already led to a lot of interesting scheduler-development work, the subsystem itself is still undergoing rapid evolution. Among other work, the ability to set up a hierarchy of sub-schedulers is approaching completion, and a longstanding incompatibility with proxy execution is coming to an end.

The sub-scheduler enqueue path

Sched_ext, as originally implemented, only allowed the installation of a single custom scheduler on any given system. It did not take long, though, for users of multi-tenant systems to request the ability to set up different schedulers for different groups of processes. The result was the addition of sub-schedulers, which associate a sched_ext scheduler with a control group; all processes within the given group will be managed by the attached scheduler. As with control groups in general, sched_ext schedulers are arranged into a hierarchy, with higher-level schedulers deciding when the lower levels are able to place processes on a CPU.

The initial sub-scheduler work was merged for the 7.1 kernel release, but it is incomplete. Specifically, sub-schedulers can handle the dispatch path — choosing which process to run next on a given CPU. Dispatch handling is hierarchical, in that a parent-group scheduler decides when to run the dispatch handler for each of its attached subgroup schedulers. Dispatching can take a process out of a dispatch queue, but it does not address how they are placed into that queue to begin with.

This series from Tejun Heo addresses the enqueue path which, naturally, turns out to have complexities of its own. The dispatch side is relatively easy; control proceeds down the scheduler hierarchy, with each level deciding which sub-scheduler below it is allowed to place tasks onto a given CPU's queue at any given time. A sub-scheduler's enqueue() callback, though, can be invoked at any time, directly from the scheduler core, when a task under its control becomes runnable. It can then (in current kernels) place that task on any CPU's dispatch queue, with the ability to preempt whatever is running there, regardless of whether the preempted task is under the same scheduler's control or not.

Control groups are meant to provide isolation between groups of processes, and the enqueue path needs to preserve that isolation. Since scheduling is a performance-critical activity, the enqueue path should, if possible, avoid calling up through the scheduler hierarchy to decide whether a given sub-scheduler is able to make a specific placement decision. The answer to that problem is a capability mechanism that allows schedulers to share some of their access with any sub-schedulers below them.

When a sub-scheduler attaches to the hierarchy, it initially has no access to any CPUs in the system. Its parent can then (with a call to scx_bpf_sub_grant()) give the new scheduler the ability to access a specific set of CPUs. That access comes in three levels: the ability to place a task on an idle CPU, the ability to put tasks into the CPU's dispatch queue for later execution, and the ability to preempt a task that is controlled by another scheduler. The root scheduler has all capabilities on all CPUs; it can then pass all or some of them down to the sub-schedulers that attach to it. A scheduler can only give capabilities to sub-schedulers that it, itself, possesses.

If a sub-scheduler tries to enqueue a task onto a CPU where it lacks the required capabilities, the task will be diverted to a special reject queue, after which it will be handed back to the same sub-scheduler's enqueue() callback with a special flag (SCX_TASK_REENQ_CAP) telling the scheduler to try again. A similar thing happens if a parent revokes capabilities previously given to a sub-scheduler: any tasks scheduled on the affected CPUs will be removed, then passed back to the sub-scheduler for a new placement.

With these changes, the sub-scheduler work is nearing completion, though there are a couple of remaining details listed by Heo in the cover letter. One is that moving a process from one control group to another will not change the sub-scheduler that controls it. The other is that it is possible to restrict (using, for example, sched_setaffinity()) a process to a set of CPUs outside of those that its sub-scheduler is able to run processes on; in that case, the process ends up being unable to run anywhere. That state of affairs will certainly improve isolation, but the owner of the affected process is unlikely to appreciate that improvement.

Those problems will have to wait for further work. Meanwhile, the patches for the enqueue path have been through five revisions (as of this writing) and are currently on track for merging into the 7.3 kernel release.

Getting along with proxy execution

Priority inversion happens when a low-priority process holds a resource needed by a higher-priority process, but the low-priority process is unable to run. Priority inheritance — letting the resource-holding process run at the waiting process's priority until it releases the resource — is a commonly used solution to the problem, but priority inheritance is not applicable to deadline scheduling. Deadline tasks do not use priorities, but they can still be blocked on resources held by other processes.

The solution in that case is proxy execution, which can be thought of as an extension of the priority-inheritance idea. With proxy execution, the waiting process's entire scheduling context is loaned to the resource-holding process, allowing it to run in the waiting process's place (and using the waiting process's resources). In the absence of proxy execution, a process that blocks on a mutex will be removed from its run queue. When proxy execution is enabled, instead, that process will remain in its run queue; when the scheduler picks it to run in a CPU, the link to the process holding the mutex will be followed, and that process will be executed in its place, using its scheduling context.

This algorithm works for the kernel's built-in scheduler, but it can create confusion when sched_ext is in use. A sched_ext scheduler, believing that it has decided which process runs, will likely be surprised to find a different process (which, perhaps, is entirely outside of its control) running instead. For this reason, proxy execution and sched_ext have been made mutually exclusive in the kernel's configuration system; a kernel can only be built with one of them enabled. If nothing else, this situation creates problems for distributors, which would rather not have to choose between two features that its users will want access to.

A potential solution to this problem can be found in this patch set from Andrea Righi and John Stultz. It makes two fundamental changes to how sched_ext works with proxy execution, the first of which is to hide most of it from sched_ext schedulers entirely. When a scheduler selects a blocked process to run, the lock holder will be transparently run instead, but the sched_ext scheduler will never see that redirection, so it will not be surprised to find a process running that it didn't schedule, and may not control at all. The waiting process's scheduling context will be used, and that process will be charged for the CPU time used by the lock holder.

During proxy execution, the waiting process's scheduling context will be moved to the CPU on which the lock holder is running. Once the lock has been released, the scheduling context is moved back, and the scheduler is invoked to re-enqueue the (now runnable) process.

Before any of this can happen, though, the scheduler must indicate to the sched_ext core that it is willing to participate in proxy execution; that is done by providing the new SCX_OPS_ENQ_BLOCKED flag at registration time. When a scheduler has registered itself that way, processes that are blocked on a mutex will be passed to the scheduler's enqueue() callback with the SCX_ENQ_BLOCKED flag set. The scheduler can then decide if it wants to handle the scheduling of the blocked task specially. The series includes a simple change to the example scx_qmap scheduler that causes it to schedule blocked tasks immediately, preempting whatever happens to be running at the time. This "intentionally unfair" policy is meant to make proxy execution easy to observe; it also shows how little is needed to make a sched_ext scheduler work with proxy execution once the infrastructure is in place.

How well proxy execution will work will depend, though, on the mix of processes in the system. If sched_ext is only in charge of some of the running processes, it will find itself unable to preempt those running outside of sched_ext, which will have a higher priority. There is a table in the changelog of this patch describing what happens in eight different situations, depending on the scheduling status of the waiting process, the lock holder, and an unrelated competing process on the CPU.

This series has been through seven revisions to date, and is aimed at merging for the 7.3 release.

Comments (none posted)