Frédéric Wang    About    Mathematics    Computer Science    Miscellaneous    Archive

MathZilla collection ported to WebExtensions

MathZilla is a collection of MathML-related add-ons for Mozilla applications. It provides nice features such as forcing native MathML rendering (e.g. on Wikipedia), using Web fonts to render MathML or providing a context menu item to copy math formulas into the clipboard.

Initially written as a single XUL overlay extension (with even binary code for the LaTeX-to-MathML converter) it grows up as a collection of restartless add-ons using bootstrapped or SDK-based extensions, following the evolution of Mozilla’s recommendations. Also, SDK-based extensions were first generated using a Python program called cfx before Mozilla recommended to switch to a JS-based replacement called jpm.

Mozilla announced some time ago that they will transition to the WebExtensions format. On the one hand this sounds bad because developers have to re-write their legacy add-ons again and actually be sure that the transition is even possible or does break anything. On the other hand it is good for long-term interoperability since e.g. Chromium browsers or Microsoft Edge support that format. My colleague Michael Catanzaro also mentioned in a recent blog post that WebExtensions are considered for Epiphany too. It is not clear what Mozilla’s plan is for Thunderbird or SeaMonkey but hopefully they will use that format too (in the past I was suggested to make the MathZilla add-ons compatible with SeaMonkey).

Recently, Mozilla announced their plans for Firefox 57 which is basically to allow only add-ons written as WebExtensions. This means I had to re-write the Mathzilla add-ons again or they will stop working at the end of the year. In general, I believe the features have been preserved although there might be some small behavior changes or minor bugs due to the WebExtensions format. Please check the GitHub bug trackers and release notes for known issues and report any other problems you find. Finally, I reorganized a bit the git repositories and add-on names. Here is the updated list (some add-ons are still being reviewed by Mozilla):

  • MathML Fonts (~2300 users) - Provide MathML fonts as Web fonts, which is useful when they can not be installed (e.g. Firefox for Android).
  • Native MathML (~1400 users) - Force MathJax/KaTeX/MediaWiki to use native MathML rendering.
  • MathML Copy (~500 users) - Add context menu items to copy a MathML formula or other annotations attached to it (e.g. LaTeX) into the clipboard.
  • TeXZilla (~500 users) - Add-on giving access to TeXZilla, a Unicode TeX-to-MathML converter.
  • MathML Font Settings (~300 users) - Add context menu items to configure MathML font settings. Note that in recent Mozilla versions the advanced font preferences menu allows to configure “Fonts for Mathematics”.
  • Presentation MathML Polyfill (~200 users) - Add support for some advanced presentation MathML features (currently using David Carlisle’s “mml3ff” XSLT stylesheet).
  • Content MathML Polyfill (~200 users) - Add support for some content MathML features (currently using David Carlisle’s “ctop” XSLT stylesheet).
  • MathML Zoom (~100 users) - Allow zooming of mathematical formulas.
  • MathML View Source (experimental) - This is a re-writing of Mozilla’s ‘view MathML source’ feature with better syntax highlighting and serialization. The idea originated from this thread.
  • Image To MathML (experimental) - Try and convert images of mathematical formulas into MathML. It has not been ported to WebExtensions yet and I do not plan to do it in the short term.

As a conclusion, I’d like to thank all the MathZilla users for their kind comments, bug reporting and financial support. The next step will probably be to ensure addons work in more browsers but that will be for another time ;-)

Mus Window System

Update 2017/02/01: The day I published this blog post, the old mus client library was removed. Some stack traces were taken during the analysis Antonio & I made in fall 2016, but they are now obsolete. I’m updating the blog in order to try and clarify things and fix possible inaccuracies.

TL; DR

Igalia has recently been working on making Chromium Desktop run natively on Ozone/Wayland thanks to support from Renesas. This is however still experimental and there is a lot of UI work to do. In order to facilitate discussion at BlinkOn7 and more specifically how browser windows are managed, we started an analysis of the Chromium Window system. An overview is provided in this blog post.

Introduction

Antonio described how we were able to get upstream Chromium running on Linux/Ozone/Wayland last year. However for that purpose the browser windows were really embedded in the “ash” environment (with additional widgets) which was itself drawn in an Ozone/Wayland window. This configuration is obviously not ideal but it at least allowed us to do some initial experiments and to present demos of Chromium running on R-Car M3 at CES 2017. If you are interested you can check our ces-demos-2017 and meta-browser repositories on GitHub. Our next goal is to have all the browser windows handled as native Ozone/Wayland windows.

In a previous blog post, I gave an overview of the architecture of Ozone. In particular, I described classes used to handle Ozone windows for the X11 or Wayland platforms. However, this was only a small part of the picture and to understand things globally one really has to take into account the classes for the Aura window system and Mus window client & server.

Window-related Classes

Class Hierarchy

A large number of C++ classes and types are involved in the chromium window system. The following diagram provides an overview of the class hierarchy. It can merely be divided into four parts:

  • In blue, the native Ozone Windows and their host class in Aura.
  • In red and orange, the Aura Window Tree.
  • In purple, the Mus Window Tree (client-side). Update 2017/02/01: These classes have been removed.
  • In green and brown, the Mus Window Tree (server-side) and its associated factories.

Windowing System

I used the following convention which is more or less based on UML:

  • Rectangles represent classes/interfaces while ellipses represent enums, types, defines etc. You can click them to access the corresponding C++ source file.
  • Inheritance is indicated by a white arrow to the base class from the derived class. For implementation of Mojo interfaces, dashed white arrows are used.
  • Other associations are represented by an arrow with open or diamond head. They mean that the origin of the arrow has a member involving one or more instances of the pointed class, you can hover over the arrow head to see the name of that class member.

Native Ozone Windows (blue)

ui::PlatformWindow is an abstract class representing a single window in the underlying platform windowing system. Examples of implementations include ui::X11WindowOzone or ui::WaylandWindow.

ui::PlatformWindow can be stored in a ui::ws::PlatformDisplayDefault which implements the ui::PlatformWindowDelegate interface too. This in turn allows to use platform windows as ui::ws::Display. ui::ws::Display also has an associated delegate class.

aura::WindowTreeHost is a generic class that hosts an embedded aura:Window root and bridges between the native window and that embedded root window. One can register instance of aura::WindowTreeHostObserver as observers. There are various implementations of aura::WindowTreeHost but we only consider aura::WindowTreeHostPlatform here.

Native ui::PlatformWindow windows are stored in an instance of aura::WindowTreeHostPlatform which also holds the gfx::AcceleratedWidget to paint on. aura::WindowTreeHostPlatform implements the ui::PlatformWindowDelegate so that it can listen events from the underlying ui::PlatformWindow.

aura::WindowTreeHostMus is a derived class of aura::WindowTreeHostPlatform that has additional logic make it usable in mus. One can listen changes from aura::WindowTreeHostMus by implementing the aura::WindowTreeHostMusDelegate interface.

Aura Windows (orange)

aura::Window represents windows in Aura. One can communicate with instances of that class using a aura::WindowDelegate. It is also possible to listen for events by registering a list of aura::WindowObserver.

aura::WindowPort defines an interface to enable aura::Window to be used either with mus via aura::WindowMus or with the classical environment via aura::WindowPortLocal. Here, we only consider the former.

WindowPortMus holds a reference to a aura::WindowTreeClient. All the changes to the former are forwarded to the latter so that we are sure they are propagated to the server. Conversely, changes received by aura::WindowTreeClient are sent back to WindowPortMus. However, aura::WindowTreeClient uses an intermediary aura::WindowMus class for that purpose to avoid that the changes are submitted again to the server.

Aura Window Tree Client (red)

aura::WindowTreeClient is an implementation of the ui::mojom::WindowManager and ui::mojom::WindowTreeClient Mojo interfaces for Aura. As in previous classes, we can associate to it a aura::WindowTreeClientDelegate or register a list of aura::WindowTreeClientObserver.

aura::WindowTreeClient also implements the aura::WindowManagerClient interface and has as an associated aura::WindowManagerDelegate. It has a set of roots aura::WindowPortMus which serve as an intermediary to the actual aura:Window instances.

In order to use an Aura Window Tree Client, you first create it using a Mojo connector, aura::WindowTreeClientDelegate as well as optional aura::WindowManagerDelegate and Mojo interface request for ui::mojom::WindowTreeClient. After that you need to call ConnectViaWindowTreeFactory or ConnectAsWindowManager to connect to the Window server and create the corresponding ui::mojom::WindowTree connection. Obviously, AddObserver can be used to add observers.

Finally, the CreateWindowPortForTopLevel function can be used to request the server to create a new top level window and associate a WindowPortMus to it.

Mus Window Tree Client (purple)

Update 2017/02/01: These classes have been removed.

ui::WindowTreeClient is the equivalent of aura:WindowTreeClient in mus. It implements the ui::mojom::WindowManager and ui::mojom::WindowTreeClient Mojo interfaces. It derives from ui::WindowManagerClient, has a list of ui::WindowTreeClientObserver and is associated to ui::WindowManagerDelegate and ui::WindowTreeClientDelegate.

Regarding windows, they are simpler than in Aura since we only have one class ui::Window class. In a similar way as Aura, we can register ui::WindowObserver. We can also add ui::Window as root windows or embedded windows of ui::WindowTreeClient.

In order to use a Mus Window Tree Client, you first create it using a ui::WindowTreeClientDelegate as well as optional ui::WindowManagerDelegate and Mojo interface request for ui::mojom::WindowTreeClient. After that you need to call ConnectViaWindowTreeFactory or ConnectAsWindowManager to connect to the Window server and create the corresponding ui::mojom::WindowTree connection. Obviously, AddObserver can be used to add observers.

Finally, the NewTopLevelWindow and NewWindow functions can be used to create new windows. This will cause the corresponding functions to be called on the connected ui::mojom::WindowTree.

Mus Window Tree Server (green and brown)

We just saw the Aura and Mus window tree clients. The ui::ws::WindowTree Mojo interface is used to for the window tree server and is implemented in ui::ws::WindowTree. In order to create window trees, we can use either the ui::ws::WindowManagerWindowTreeFactory or the ui::ws::WindowTreeFactory which are available through the corresponding ui::mojom::WindowManagerWindowTreeFactory and ui::mojom::WindowTreeFactory Mojo interfaces.

ui::ws::WindowTreeHostFactory implements the ui::ws::WindowTreeHostFactory Mojo interface. It allows to create ui::mojom::WindowTreeHost, more precisely ui::ws::Display implementing that interface. Under the hood, ui::ws::Display created that way actually have an associated ws::DisplayBinding creating a ui::ws::WindowTree. Of course, the display also has a native window associated to it.

Mojo Window Tree APIs (pink)

Update 2017/02/01: The Mus client has been removed. This section only applies to the Aura client.

As we have just seen in previous sections the Aura and Mus window systems are very similar and in particular implement the same Mojo ui::mojom::WindowManager and ui::mojom::WindowTreeClient interfaces. In particular:

  • The ui::mojom::WindowManager::WmCreateTopLevelWindow function can be used to create a new top level window. This results in a new windows (aura::Window or ui::Window) being added to the set of embedded windows.

  • The ui::mojom::WindowManager::WmNewDisplayAdded function is invoked when a new display is added. This results in a new window being added to the set of window roots (aura::WindowMus or ui::Window).

  • The ui::mojom::WindowTreeClient::OnEmbed function is invoked when Embed is called on the connected ui::mojom::WindowTree. The window parameter will be set as the window root (aura:WindowMus or ui:Window). Note that we assume that the set of roots is empty when such an embedding happens.

Note that for Aura Window Tree Client, the creation of new aura::WindowMus window roots implies the creation of the associated aura::WindowTreeHostMus and aura::WindowPortMus. The creation of the new aura:Window window by aura::WmCreateTopLevelWindow is left to the corresponding implementation in aura::WindowManagerDelegate.

On the server side, the ui::ws::WindowTree implements the ui::mojom::WindowTree mojo interface. In particular:

  • The ui::mojom::WindowTree::Embed function can be used to embed a WindowTreeClient at a given window. This will result of OnEmbed being called on the window tree client.

  • The ui::mojom::WindowTree::NewWindow function can be used to create a new window. It will call OnChangeCompleted on the window tree client.

  • The ui::mojom::WindowTree::NewTopLevelWindow function can be used to request the window manager to create a new top level window. It will call OnTopLevelCreated (or OnChangeCompleted in case of failure) on the window tree client.

Analysis of Window Creation

Chrome/Mash

In our project, we are interested in how Chrome/Mash windows are created. When the chrome browser process is launched, it takes the regular content/browser startup path. That means in chrome/app/chrome_main.cc it does not take the ::MainMash path, but instead content::ContentMain. This is the call stack of Chrome window creation within the mash shell:

#1 0x7f7104e9fd02 ui::WindowTreeClient::NewTopLevelWindow()
#2 0x5647d0ed9067 BrowserFrameMus::BrowserFrameMus()
#3 0x5647d09c4a47 NativeBrowserFrameFactory::Create()
#4 0x5647d096bbc0 BrowserFrame::InitBrowserFrame()
#5 0x5647d0877b8a BrowserWindow::CreateBrowserWindow()
#6 0x5647d07d5a48 Browser::Browser()
...

Update 2017/02/01: This has changed since the time when the stack trace was taken. The BrowserFrameMus constructor now creates a views:DesktopNativeWidgetAura and thus an aura::Window.

The outtermost window created when one executes chrome --mash happens as follows. When the UI service starts (see Service::OnStart in services/ui/service.cc), an instance of WindowServer is created. This creations triggers the creation of a GpuServiceProxy instance. See related snippets below:

void Service::OnStart() {
  (...)
  // Gpu must be running before the PlatformScreen can be initialized.
  window_server_.reset(new ws::WindowServer(this));
  (..)
}

WindowServer::WindowServer(WindowServerDelegate* delegate)
: delegate_(delegate),
  (..)
  gpu_proxy_(new GpuServiceProxy(this)),
  (..)
{ }

GpuServiceProxy, then, starts off the GPU service initialization. By the time the GPU connection is established, the following callback is called: GpuServiceProxy::OnInternalGpuChannelEstablished. This calls back to WindowServer::OnGpuChannelEstablished, which calls Service::StartDisplayInit. The later schedules a call to PlatformScreenStub::FixedSizeScreenConfiguration. This finally triggers the creation of an Ozone top level window via an ui::ws::Display:

#0 base::debug::StackTrace::StackTrace()
#1 ui::(anonymous namespace)::OzonePlatformX11::CreatePlatformWindow()
#2 ui::ws::PlatformDisplayDefault::Init()
#3 ui::ws::Display::Init()
#4 ui::ws::DisplayManager::OnDisplayAdded()
#5 ui::ws::PlatformScreenStub::FixedSizeScreenConfiguration()
(..)

On the client-side, the addition of a new display will cause aura::WindowTreeClient::WmNewDisplayAdded to be called and will eventually lead to the creation of an aura::WindowTreeHostMus with the corresponding display id. As shown on the graph, this class derives from aura::WindowTreeHost and hence instantiates an aura::Window.

Mus demo

The Chromium source code contains a small mus demo that is also useful do experiments. The demo is implemented via a class with the following inheritance:

class MusDemo : public service_manager::Service,
                public aura::WindowTreeClientDelegate,
                public aura::WindowManagerDelegate

i.e. it can be started as Mus service and handles some events from aura:WindowTreeClient and aura::WindowManager.

When the demo service is launched, it creates a new Aura environment and an aura::WindowTreeClient, it does the Mojo request to create an ui::WindowTree and then performs the connection:

#0 aura::WindowTreeClient::ConnectAsWindowManager()
#1 ui::demo::MusDemo::OnStart()
#2 service_manager::ServiceContext::OnStart()

Similarly to Chrome/Mash, a new display and its associated native window is created by the UI:

#0 ui::ws::WindowTree::AddRootForWindowManager()
#1 ui::ws::Display::CreateWindowManagerDisplayRootFromFactory()
#2 ui::ws::Display::InitWindowManagerDisplayRoots()
#3 ui::ws::PlatformDisplayDefault::OnAcceleratedWidgetAvailable()
#4 ui::X11WindowBase::Create()
#5 ui::(anonymous namespace)::OzonePlatformX11::CreatePlatformWindow()
#6 ui::ws::PlatformDisplayDefault::Init()
#7 ui::ws::Display::Init()
#8 ui::ws::DisplayManager::OnDisplayAdded()

The Aura Window Tree Client listens the changes and eventually aura::WindowTreeClient::WmNewDisplayAdded is called. It then informs its MusDemo delegate so that the display is added to the screen display list

#0 DisplayList::AddDisplay
#1 ui::demo::MusDemo::OnWmWillCreateDisplay
#2 aura::WindowTreeClient::WmNewDisplayAddedImpl
#3 aura::WindowTreeClient::WmNewDisplayAdded

…and the animated bitmap window is added to the Window Tree:

#0  ui::demo::MusDemo::OnWmNewDisplay
#1  aura::WindowTreeClient::WmNewDisplayAddedImpl
#2  aura::WindowTreeClient::WmNewDisplayAdded

Update 2017/02/01: Again, aura::WindowTreeClient::WmNewDisplayAdded will lead to the creation of an aura Window.

Conclusion

Currently, both Chrome/Mash or Mus demo create a ui::ws::Display containing a native Ozone window. The Mus demo class uses the Aura WindowTreeClient to track the creation of that display and pass it to the Mus Demo class. The actual chrome browser window is created inside the native Mash window using a Mus WindowTreeClient. Again, this is not what we want since each chrome browser should have its own native window. It will be interesting to first experiment multiple native windows with the Mus demo.

Update 2017/02/01: aura::WindowTreeClient is now always used to create Aura Windows for both external windows (corresponding to a display & a native window) and internal windows. However, we still want all chrome browser to be external windows.

Igalia Logo
Renesas Logo

It seems that the Ozone/Mash work has been mostly done with ChromeOS in mind and will probably need some adjustments to make things work on desktop Linux. We are very willing to discuss this and other steps for Desktop Chrome Wayland/Ozone in details with Google engineers. Antonio will attend BlinkOn7 next week and will be very happy to talk with you so do not hesitate to get in touch!

¡Igalia is hiring!

If you read this blog, you probably know that I joined Igalia early this year, where I have been involved in projects related to free software and web engines. You may however not be aware that Igalia has a flat & cooperative structure where all decisions (projects, events, recruitments, company agreements etc) are voted by members of an assembly. In my opinion such an organization allows to take better decisions and to avoid frustrations, compared to more traditional hierarchical organizations.

After several months as a staff, I finally applied to become an assembly member and my application was approved in November! Hence I attended my first assembly last week where I got access to all the internal information and was also able to vote… In particular, we approved the opening of two new job positions. If you are interested in state-of-the-art free software projects and if you are willing to join a company with great human values, you should definitely consider applying!

STIX Two in Gecko and WebKit

On the 1st of December, the STIX Fonts project announced the release of STIX 2. If you never heard about this project, it is described as follows:

The mission of the Scientific and Technical Information Exchange (STIX) font creation project is the preparation of a comprehensive set of fonts that serve the scientific and engineering community in the process from manuscript creation through final publication, both in electronic and print formats.

This sounds a very exciting goal but the way it has been achieved has made the STIX project infamous for its numerous delays, for its poor or confusing packaging, for delivering math fonts with too many bugs to be usable, for its lack of openness & communication, for its bad handling of third-party feedback & contribution

Because of these laborious travels towards unsatisfactory releases, some snarky people claim that the project was actually named after Styx (Στύξ) the river from Greek mythology that one has to cross to enter the Underworld. Or that the story of the project is summarized by Baudelaire’s verses from L’Irrémédiable:

Une Idée, une Forme, un Être
Parti de l’azur et tombé
Dans un Styx bourbeux et plombé
Où nul œil du Ciel ne pénètre ;

More seriously, the good news is that the STIX Consortium finally released text fonts with a beautiful design and a companion math font that is usable in math rendering engines such as Word Processors, LaTeX and Web Engines. Indeed, WebKit and Gecko have supported OpenType-based MathML layout for more than three years (with recent improvements by Igalia) and STIX Two now has correct OpenType data and metrics!

Of course, the STIX Consortium did not address all the technical or organizational issues that have made its reputation but I count on Khaled Hosny to maintain his more open XITS fork with enhancements that have been ignored for STIX Two (e.g. Arabic and RTL features) or with fixes of already reported bugs.

As Jacques Distler wrote in a recent blog post, OS vendors should ideally bundle the STIX Two fonts in their default installation. For now, users can download and install the OTF fonts themselves. Note however that the STIX Two archive contains WOFF and WOFF2 fonts that page authors can use as web fonts.

I just landed patches in Gecko and WebKit so that future releases will try and find STIX Two on your system for MathML rendering. However, you can already do the proper font configuration via the preference menu of your browser:

  • For Gecko-based applications (e.g. Firefox, Seamonkey or Thunderbird), go to the font preference and select STIX Two Math as the “font for mathematics”.
  • For WebKit-based applications (e.g. Epiphany or Safari) add the following rule to your user stylesheet: math { font-family: "STIX Two Math"; }.

Finally, here is a screenshot of MathML formulas rendered by Firefox 49 using STIX Two:

Screenshot of MathML formulas rendered by Firefox using STIX 2

And the same page rendered by Epiphany 3.22.3:

Screenshot of MathML formulas rendered by Epiphany using STIX 2

Chromium on R-Car M3 & AGL/Wayland

As my fellow igalian Antonio Gomes explained on his blog, we have recenly been working on making the master branch of Chromium run on Linux Desktop using the upstream Ozone/Wayland backend. This effort is supported by Renesas and they were interested to rely on these recent developments to showcase Chromium running on the latest generation of their R-Car systems-on-chip for automotive applications. Ideally, they were willing to see this happening for the Automotive Grade Linux distribution.

Igalia Logo
Renesas Logo

Luckily, support for R-Car M3 was already available in the development branch of AGL and the AGL instructions for R-Car M2 actually worked pretty well for M3 too mutatis mutandis. There is also an OpenEmbedded/Yocto BSP layer for Chromium but unfortunately Wayland is only supported up to m48 (using the Ozone backend from Intel’s fork) and X11 is only supported up to m52. Hence we created a (for-now-private) fork of meta-browser that:

  • Uses the latest development version of Chromium, in particular the Linux/Ozone/Mash/Wayland support Igalia has been working on recently.
  • Relies on the new GN build system rather than the deprecated GYP one.
  • Supports the ARM64 architecture instead of just ARM32.
  • Installs more resources such as Mojo service manifests.
  • Applies more patches (e.g. build fixes or the workaround for issue 2485673002).

Renesas also provided us some R-Car M3 boards. I received my package this week and hence was able to start playing with it on Wednesday. Again, the AGL instructions for R-Car M2 apply well to M3 and the elinux page is very helpful to understand the various parts of the board. After having started AGL on the board and opened a Weston terminal (using mouse & keyboard as on the video or using ssh) I was able to run chromium via the following command:

/usr/bin/chromium/chrome --mash --ozone-plaform=wayland \
                         --user-data-dir=/tmp/user-data-dir \
                         --no-sandbox

The first line is the usual command currently used to execute Ozone/Mash/Wayland. The second line works around the fact that the AGL demo is running as root. The third line disables sandbox because the deprecated setuid sandbox does not work with Ozone and the newer user namespaces sandbox is not supported by the AGL kernel.

The following video a gives an overview of the hardware preparation as well as some basic tests with bouncyballs.org and webengineshackfest.org:

You can see that chromium runs relatively smoothly but we will do more testing to confirm these initial experiments. Also, we find the general issues with Linux/Ozone/Mash (internal VS external windows, no fullscreen, keyboard restricted to US layout, unwanted ChromeOS widgets etc) but we expect to continue to collaborate with Google to fix these bugs!