Frédéric Wang    About    Mathematics    Computer Science    Miscellaneous    Archive

¡Igalia is hiring!

If you read this blog, you probably know that I joined Igalia early this year, where I have been involved in projects related to free software and web engines. You may however not be aware that Igalia has a flat & cooperative structure where all decisions (projects, events, recruitments, company agreements etc) are voted by members of an assembly. In my opinion such an organization allows to take better decisions and to avoid frustrations, compared to more traditional hierarchical organizations.

After several months as a staff, I finally applied to become an assembly member and my application was approved in November! Hence I attended my first assembly last week where I got access to all the internal information and was also able to vote… In particular, we approved the opening of two new job positions. If you are interested in state-of-the-art free software projects and if you are willing to join a company with great human values, you should definitely consider applying!

STIX Two in Gecko and WebKit

On the 1st of December, the STIX Fonts project announced the release of STIX 2. If you never heard about this project, it is described as follows:

The mission of the Scientific and Technical Information Exchange (STIX) font creation project is the preparation of a comprehensive set of fonts that serve the scientific and engineering community in the process from manuscript creation through final publication, both in electronic and print formats.

This sounds a very exciting goal but the way it has been achieved has made the STIX project infamous for its numerous delays, for its poor or confusing packaging, for delivering math fonts with too many bugs to be usable, for its lack of openness & communication, for its bad handling of third-party feedback & contribution

Because of these laborious travels towards unsatisfactory releases, some snarky people claim that the project was actually named after Styx (Στύξ) the river from Greek mythology that one has to cross to enter the Underworld. Or that the story of the project is summarized by Baudelaire’s verses from L’Irrémédiable:

Une Idée, une Forme, un Être
Parti de l’azur et tombé
Dans un Styx bourbeux et plombé
Où nul œil du Ciel ne pénètre ;

More seriously, the good news is that the STIX Consortium finally released text fonts with a beautiful design and a companion math font that is usable in math rendering engines such as Word Processors, LaTeX and Web Engines. Indeed, WebKit and Gecko have supported OpenType-based MathML layout for more than three years (with recent improvements by Igalia) and STIX Two now has correct OpenType data and metrics!

Of course, the STIX Consortium did not address all the technical or organizational issues that have made its reputation but I count on Khaled Hosny to maintain his more open XITS fork with enhancements that have been ignored for STIX Two (e.g. Arabic and RTL features) or with fixes of already reported bugs.

As Jacques Distler wrote in a recent blog post, OS vendors should ideally bundle the STIX Two fonts in their default installation. For now, users can download and install the OTF fonts themselves. Note however that the STIX Two archive contains WOFF and WOFF2 fonts that page authors can use as web fonts.

I just landed patches in Gecko and WebKit so that future releases will try and find STIX Two on your system for MathML rendering. However, you can already do the proper font configuration via the preference menu of your browser:

  • For Gecko-based applications (e.g. Firefox, Seamonkey or Thunderbird), go to the font preference and select STIX Two Math as the “font for mathematics”.
  • For WebKit-based applications (e.g. Epiphany or Safari) add the following rule to your user stylesheet: math { font-family: "STIX Two Math"; }.

Finally, here is a screenshot of MathML formulas rendered by Firefox 49 using STIX Two:

Screenshot of MathML formulas rendered by Firefox using STIX 2

And the same page rendered by Epiphany 3.22.3:

Screenshot of MathML formulas rendered by Epiphany using STIX 2

Chromium on R-Car M3 & AGL/Wayland

As my fellow igalian Antonio Gomes explained on his blog, we have recenly been working on making the master branch of Chromium run on Linux Desktop using the upstream Ozone/Wayland backend. This effort is supported by Renesas and they were interested to rely on these recent developments to showcase Chromium running on the latest generation of their R-Car systems-on-chip for automotive applications. Ideally, they were willing to see this happening for the Automotive Grade Linux distribution.

Igalia Logo
Renesas Logo

Luckily, support for R-Car M3 was already available in the development branch of AGL and the AGL instructions for R-Car M2 actually worked pretty well for M3 too mutatis mutandis. There is also an OpenEmbedded/Yocto BSP layer for Chromium but unfortunately Wayland is only supported up to m48 (using the Ozone backend from Intel’s fork) and X11 is only supported up to m52. Hence we created a (for-now-private) fork of meta-browser that:

  • Uses the latest development version of Chromium, in particular the Linux/Ozone/Mash/Wayland support Igalia has been working on recently.
  • Relies on the new GN build system rather than the deprecated GYP one.
  • Supports the ARM64 architecture instead of just ARM32.
  • Installs more resources such as Mojo service manifests.
  • Applies more patches (e.g. build fixes or the workaround for issue 2485673002).

Renesas also provided us some R-Car M3 boards. I received my package this week and hence was able to start playing with it on Wednesday. Again, the AGL instructions for R-Car M2 apply well to M3 and the elinux page is very helpful to understand the various parts of the board. After having started AGL on the board and opened a Weston terminal (using mouse & keyboard as on the video or using ssh) I was able to run chromium via the following command:

/usr/bin/chromium/chrome --mash --ozone-plaform=wayland \
                         --user-data-dir=/tmp/user-data-dir \
                         --no-sandbox

The first line is the usual command currently used to execute Ozone/Mash/Wayland. The second line works around the fact that the AGL demo is running as root. The third line disables sandbox because the deprecated setuid sandbox does not work with Ozone and the newer user namespaces sandbox is not supported by the AGL kernel.

The following video a gives an overview of the hardware preparation as well as some basic tests with bouncyballs.org and webengineshackfest.org:

You can see that chromium runs relatively smoothly but we will do more testing to confirm these initial experiments. Also, we find the general issues with Linux/Ozone/Mash (internal VS external windows, no fullscreen, keyboard restricted to US layout, unwanted ChromeOS widgets etc) but we expect to continue to collaborate with Google to fix these bugs!

Analysis of Ozone Wayland

Introduction

In the past two months, I have been working with my colleague Antonio Gomes on a Chromium project supported by Renesas. The goal is to have Chrome running on Wayland with accelerated rendering happening in a separate GPU process. Intel has done a great job to develop a specific Wayland platform for Ozone but unfortunately the code only works for older releases of Chromium and is not fully upstreamed yet. In theory, easy out-of-tree platforms was one of the guiding principles of Ozone but in practice we had to fix many build and run time errors for Ozone, even though we were working upstream. More generally, it is very challenging to keep in sync with all the refactoring work happening upstream and some work is definitely required to align Intel’s code with Google’s goals.

To summarize the situation briefly, if we follow Intel’s approach then the currently upstreamed Wayland code only compiles for ChromeOS (i.e. target_os="chromeos" in your build config) not for standard Linux build of Chrome. You can only have accelerated rendering at the price of removing separation of UI/GPU processes (i.e. via the --in-process-gpu switch). If you keep the default behavior of having the UI and GPU components running in separate processes then accelerated rendering fails. A fallback to software rendering is then attempted but it in turn fails in gpu_process_transport_factory.cc.

When Antonio joined Igalia, he did some experiments on Ozone/Wayland and wrote a quick workaround to make the software rendering fallback work when UI and GPU are in separate processes. He was also able to run standard Linux build of Chrome on Wayland by upstreaming more code from Intel. He also noticed that in Google’s code, only GL drawing is happening in the GPU component (i.e. Wayland objects are owned by the UI, contrary to Intel’s approach) which is the reason why accelerated rendering fails in separate-process mode. After discussion with Google developers it however became clear that they would prefer a different approach that is consistent with two features they are working on:

  • Mus-ash, a project to separate the window management and shell functionality of ash from the chrome browser process.

  • Mojo, a new API for inter-process communication.

During the project, we were able to get chrome running on Wayland using the Mus code path, either on ChromeOS or on standard Linux build. For that code path, GPU and UI components are currently running in the same process so as discussed above accelerated rendering works for Wayland. However, the plan for mus is to move these two components into separate processes and hence we need to adapt the Wayland code in order to allow communication between the GPU (doing GL drawings) and the UI (owning Wayland objects).

I’ll let Antonio describe more precisely on his blog the work we have been doing to get chrome --mash running on Wayland. In this blog post, I’m aligning with Google’s goal and hence I’m focusing on this Mus code path. I’m going to give a quick overview of the structure of Ozone and more specifically what is used by the Wayland platform to perform accelerated rendering.

Ozone Architecture

Ozone Platform

OzonePlatform is the main Ozone interface used to instantiate and initialize an Ozone platform (X11, Wayland, DRM/GBM…). It provides factory getters for helper classes (SurfaceFactoryOzone, PlatformWindow…).

The goal is to have OzonePlatform::InitializeForUI and OzonePlatform::InitializeForGPU called from different processes as well as two different Mojo connectors for the UI and GPU services respectively. However at the time of writing, the two components are only in different threads in the same process. There is also only one Mojo connector available: The one for the service:ui service, passed to OzonePlatform::InitializeForUI.

Two important implementations to consider in this project are OzonePlatformWayland (the one we work on) and OzonePlatformGbm (the one that seems the best maintained and tested upstream).

Native Display Delegate

NativeDisplayDelegate is used to perform display configuration. The DRM/GBM platform has its own DrmNativeDisplayDelegate implementation. For the Wayland platform, we do not actually need real display devices, so we just do as for X11 and use the FakeDisplayDelegate class. See Issue 2389053003.

Window

AcceleratedWidget is just a platform-specific object representing a surface on which compositors can paint pixels.

PlatformWindow represents a single window in the underlying platform windowing system with the usual property: It can be minimized, maximized, closed, put in fullscreen, etc

PlatformWindowDelegate. This is just a delegate to which the PlatformWindow sends events like e.g. OnBoundsChanged.

The implementations of PlatformWindow used for the Wayland and DRM/GBM platforms are respectively WaylandWindow and DrmWindow. At the moment, several features in WaylandWindow are not fully implemented but the minimal code to paint content into the window is present.

Surface and OpenGL

SurfaceFactoryOzone provides surface to be used to paint on a window. The implementations for the Wayland and DRM/GBM platforms are respectively WaylandSurfaceFactory and GbmSurfaceFactory. There are two options:

  • Accelerated drawing (GL path). This is done via a GLOzone instance returned by SurfaceFactoryOzone::GetGLOzone
  • Software Drawing (Skia). This is done via a SurfaceOzoneCanvas instance returned by SurfaceFactoryOzone::CreateCanvasForWidget.

In this project, we focus on accelerated rendering and on EGL, so we consider the GLOzoneEGL class. The following virtual pure functions must be implemented in derived class:

  • GLOzoneEGL::LoadGLES2Bindings, performing the GL initalization. In general, the default works well.
  • GLOzoneEGL::CreateOffscreenGLSurface returning an offscreen GLSurface of the specified dimension. It seems that it is mostly needed to create a dummy zero-dimensional offscreen surface during initialization. Hence the generic SurfacelessEGL should work fine. See Issue 2387063002.
  • GLOzoneEGL::GetNativeDisplay returns the EGL display connection to use.
  • GLOzoneEGL::CreateViewGLSurface returns a new GLSurface associated to a gfx::AcceleratedWidget.

SurfaceFactoryOzone also provides functions to create NativePixmap, which represents a buffer that can be directly imported via GL for rendering, or exported via dma-buf fds. The DRM/GBM platform implements it and uses GbmPixmap. For now, such pixmap objects are not needed by the Wayland platform so the SurfaceFactoryOzone function members are not implemented.

The instances of GLSurface returned by CreateViewGLSurface for the Wayland and DRM/GBM platforms are respectively GLSurfaceWayland and GbmSurface. GLSurfaceWayland is just a gl::NativeViewGLSurfaceEGL associated to a window created by wl_egl_window_create. GbmSurface instead provides surface-like semantics by deriving from GbmSurfaceless itself deriving from gl::SurfacelessEGL. Internally, a framebuffer is bound automatically for GL drawing in the GPU and the result are exported to the UI via pixmaps.

Wayland Platform

Wayland Connection

WaylandConnection is a class specific to the Wayland platform that helps to instantiate all the objects necessary to communicate with the Wayland display server.

It also manages a map from gfx::AcceleratedWidget instances to WaylandWindow instances that you can modify with the public GetWindow, AddWindow and RemoveWindow function members.

Wayland Platform Initialization

  • OzonePlatformWayland::InitializeUI is called from the UI thread. It creates instances of WaylandConnection and WaylandSurfaceFactory as members of OzonePlatformWayland. The Mojo connection of the service:ui service is discarded.

  • OzonePlatformWayland::InitializeGPU in the same process but a different thread. The WaylandConnection and WaylandSurfaceFactory members previously created are hence accessible from the GPU thread too. Nothing is done by this initialization function.

Wayland Window

A WaylandWindow is tied to a WaylandConnection and takes care of registering and unregistering itself to that WaylandConnection. It is merely a wrapper to native Wayland surfaces (wl_surface and xdg_surface) with additional window bounds. It also maintains communication with the PlatformWindowDelegate.

Wayland Surface and OpenGL

Currently, the constructor of WaylandSurfaceFactory receives the WaylandConnection. Because we want the UI process to hold the WaylandConnection and the GPU process to use WaylandSurfaceFactory for the GL rendering, the current setup will not work after mus is split. Instead, we should pass it the Mojo connector of the service:gpu service in order to indirectly communicate with the service:ui service (for testing purpose, we can for now just use the Mojo connector of the service:ui service passed to OzonePlatformWayland::InitializeUI).

The WaylandSurfaceFactory::CreateCanvasForWidget function and the WaylandCanvasSurface instance created require the WaylandConnection. However, they are used for software rendering so we can ignore them for now.

GLOzoneEGL::LoadGLES2Bindings and GLOzoneEGL::CreateOffscreenGLSurface do not seem fundamental here and they do not need any Wayland-specific code. Hence we can probably just keep the current default implementations.

At the moment GLOzoneEGLWayland::GetNativeDisplay just returns a native display provided by the WaylandConnection. It seems that we will not be able to do so when the WaylandConnection is no longer available on the GPU side. Probably we should just do like GLOzoneEGLGbm and return EGL_DEFAULT_DISPLAY.

The remaining GLOzoneEGLWayland::CreateViewGLSurface uses WaylandConnection::GetWindow to retrieve the WaylandWindow associated to AcceleratedWidget to draw on. This window provides a wl_surface and sizes that can be passed to wl_egl_window_create to create an egl_window. Then as said in the previous paragraph, a GLSurfaceWayland instance is created which is merely a gl::NativeViewGLSurfaceEGL associated to the egl_window.

Conclusion

In this blog post, an overview of the Ozone Architecture was provided, with focus on the Wayland platform. It also contains an analysis of how we could get accelerated rendering in a separate GPU process aligned with Google’s goals (Mus+ash and Mojo) and hence have it well-integrated with upstream code.

The main problem is that the GLOzoneEGLWayland code is very tied to the Wayland native objects (connection, surface, window…). We should instead only provide a Mojo connector to the GLOzoneEGLWayland class and hence to the GLSurfaceWayland class it constructs. Instances of WaylandWindow and WaylandConnector will only live on the UI side while the GL classes will live on the GPU side.

The GLSurfaceWayland should be rewritten to derive from SurfacelessEGL instead of NativeViewGLSurfaceEGL. We would then follow what is done in GbmSurface to provide some surface-like semantics but without the need to have a real egl_window object. Under the hood, the GL drawings will be performed on a framebuffer.

We should then find a way to export those framebuffers to the UI component via the Mojo connector. Again, following the DRM/GBM code with this NativePixmap objects seems the right option. At the end, the UI would convert the buffers into wl_buffers that can finally be attach to the wl_surface of the WaylandWindow.

During the past two months we have been maintaining the upstream Ozone code and made the Wayland platform work again. Try bots now build and run tests for the Wayland platform and we expect that such continuous testing will make the whole thing more robust. We have also been working on making Linux desktop work with the Mus+ash code path and started encouraging experiments of Mojo communication between the UI and GPU components of the Wayland platform.

Igalia Logo
Renesas Logo

It is really great to work with Antonio on this project and we are looking forward to continuing the collaboration on this with Google and Ozone developers. Last but not least, I would like to thank Renesas for supporting Igalia in this work to add Wayland support to chromium.

Fonts at the Web Engines Hackfest 2016

Last week I travelled to Galicia for one of the regular gatherings organized by Igalia. It was a great pleasure to meet again all the Igalians and friends. Moreover, this time was a bit special since we celebrated our 15th anniversary :-)

Igalia Summit October 2016
Photo by Alberto Garcia licensed under CC BY-SA 2.0.

I also attended the third edition of the Web Engines Hackfest, sponsored by Igalia, Collabora and Mozilla. This year, we had various participants from the Web Platform including folks from Apple, Collabora, Google, Igalia, Huawei, Mozilla or Red Hat. For my first hackfest as an Igalian, I invited some experts on fonts & math rendering to collaborate on OpenType MATH support HarfBuzz and its use in math rendering engines. In this blog post, I am going to focus on the work I have made with Behdad Esfahbod and Khaled Hosny. I think it was again a great and productive hackfest and I am looking forward to attending the next edition!

OpenType MATH in HarfBuzz

Web Engines Hackfest, main room
Photo by @webengineshackfest licensed under CC BY-SA 2.0.

Behdad gave a talk with a nice overview of the work accomplished in HarfBuzz during ten years. One thing appearing recently in HarfBuzz is the need for APIs to parse OpenType tables on all platforms. As part of my job at Igalia, I had started to experiment adding support for the MATH table some months ago and it was nice to have Behdad finally available to review, fix and improve commits.

When I talked to Mozilla employee Karl Tomlinson, it became apparent that the simple shaping API for stretchy operators proposed in my blog post would not cover all the special cases currently implemented in Gecko. Moreover, this shaping API is also very similar to another one existing in HarfBuzz for non-math script so we would have to decide the best way to share the logic.

As a consequence, we decided for now to focus on providing an API to access all the data of the MATH table. After the Web Engines Hackfest, such a math API is now integrated into the development repository of HarfBuzz and will available in version 1.3.3 :-)

MathML in Web Rendering Engines

Currently, several math rendering engines have their own code to parse the data of the OpenType MATH table. But many of them actually use HarfBuzz for normal text shaping and hence could just rely on the new math API the math rendering too. Before the hackfest, Khaled already had tested my work-in-progress branch with libmathview and I had done the same for Igalia’s Chromium MathML branch.

MathML Fraction parameters test
MathML test for OpenType MATH Fraction parameters in Gecko, Blink and WebKit.

Once the new API landed into HarfBuzz, Khaled was also able to use it for the XeTeX typesetting system. I also started to experiment this for Gecko and WebKit. This seems to work pretty well and we get consistent results for Gecko, Blink and WebKit! Some random thoughts:

  • The math data is exposed through a hb_font_t which contains the text size. This means that the various values are now directly resolved and returned as a fixed-point number which should allow to avoid rounding errors we may currently have in Gecko or WebKit when multiplying by float factors.
  • HarfBuzz has some magic to automatically handle invalid offsets and sizes that greatly simplifies the code, compared to what exist in Gecko and WebKit.
  • Contrary to Gecko’s implementation, HarfBuzz does not cache the latest result for glyph-specific data. Maybe we want to keep that?
  • The WebKit changes were tested on the GTK port, where HarfBuzz is enabled. Other ports may still need to use the existing parsing code from the WebKit tree. Perhaps Apple should consider adding support for the OpenType MATH table to CoreText?

Brotli/WOFF2/OTS libraries

Web Engines Hackfest, main room
Photo by @webengineshackfest licensed under CC BY-SA 2.0.

We also updated the copies of WOFF2 and OTS libraries in WebKit and Gecko respectively. This improves one requirement from the WOFF2 specification and allows to pass the corresponding conformance test.

Gecko, WebKit and Chromium bundle their own copy of the Brotli, WOFF2 or OTS libraries in their source repositories. However:

  • We have to use more or less automated mechanisms to keep these bundled copies up-to-date. This is especially annoying for Brotli and WOFF2 since they are still in development and we must be sure to always integrate the latest security fixes. Also, we get compiler warnings or coding style errors that do not exist upstream and that must be disabled or patched until they are fixed upstream and imported again.

  • This obviously is not an optimal sharing of system library and may increase the size of binaries. Using shared libraries is what maintainers of Linux (or other FLOSS systems) generally ask and this was raised during the WebKitGTK+ session. Similarly, we should really use the system Brotli/WOFF2 bundled in future releases of Apple’s operating systems.

There are several issues that make hard for package maintainers to provide these libraries: no released binaries or release tags, no proper build system to generate shared libraries, use of git submodule to include one library source code into another etc Things have gotten a bit better for Brotli and I was able to tweak the CMake script to produce shared libraries. For WOFF2, issue 40 and issue 49 have been inactive but hopefully these will be addressed in the future…