Lately, I experimented a bit with VSTGUI again and the result is a running Kawai K1, directly in your web browser 😃

TL;DR if you want to try it: Click Here

🛑Needs a decent browser that supports web assembly! 🛑

If you want to know more about the implementation details, continue reading below.

This idea evolved while I thought about how to release my other project, Heat Synthesizer as a native VST/AU including a UI. At the moment, it is an Android app with a small proxy VST for Windows only. The reason for not having a UI in the VST version is that I created it for Android using the native Android UI system in Java. Not an optimal solution to port it to different platforms.

To be able to port Heat Synthesizer to VST & AU, I was in need of a UI system that supports a wide range of platforms:

• Android – Obviously I don’t want to drop support here so this is a must have
• VST on Windows – I want to use my synthesizer directly in my DAW that I personally use, without the need to fire up my Android tablet to edit sounds
• AU/VST on Mac – Would be beneficial if I were able to release it there, too

The Heat Synthesizer engine code compiles just fine for a wide range of platforms. I even had it running on an iPad already many years ago, just for testing purposes. But having a UI framework that covers all these platforms is not easy to find.

Actually, I recreated the UI in Unity3D some years ago. I thought this would be a good candidate to be more flexible with regards to targeting different platforms. Which is true for both iOS and Android, but Unity3D even nowadays lacks embedding support into runtime libraries so bringing it to VST and AU plugins wouldn’t be possible.

For the K1v, I’ve chosen a different strategy right from the start. Just in case you missed my development blog so far, K1v is created in C++ and the UI is made with VSTGUI. It’s a UI system by Steinberg, optimized for audio software. VSTGUI is open source and has become quite flexible throughout the years. It is no longer tightly bound any kind of existing VST SDK but also works with AU, VST3 or even in standalone mode.

What I came up with was the question if it was possible to create a fork of VSTGUI that supports Android. Knowing that I have to redo the Heat Synthesizer UI completely from scratch, my K1v project came in very handy here. K1v is a VSTGUI-based project that already compiles on a large variety of platforms (Win, Mac, Linux, both 32 and 64 bit each) that I could play around with.

K1v in Standalone Mode

Obviously, the first step was to create a standalone app of Nils‘ K1v. An Android app would be a „Standalone“ app, not a plugin. That step was pretty much straightforward and was finished after one evening.

The internal structure of K1v is that there is a generic „K1Plugin“ and a „K1Editor“ and these two are then wrapped for the different plugin platforms such as VST2, VST3 (unreleased yet) and AU.

I created an executable that opens a VSTGUI-based standalone window, embedded the K1v editor and created the K1v plugin instance.

Audio Output? MIDI?

To test the standalone app, I needed an audio output and an audio processing loop. Luckily, I solved this years ago as part of the Heat Synthesizer project. There is a simple audio output interface class with a lot of different implementations:

• OpenSLES – Used on Android
• Jack – Used for Samsung Soundcamp on Android. Unfortunately they stopped development and dropped support, but this implementation might come in handy on Linux in the future
• OpenAL – I used this for testing purposes as the SDK is very small and I already knew it from game development
• PortAudio – I added this some years ago to have lower latency on Windows
• CoreAudio – Used to test on Mac & iOS

I opted for PortAudio and OpenAL for now as I have been able to test them directly on Windows.

As for MIDI, there is just no support included. To test that everything works, I injected a Note ON event on Note C3 at startup, should be enough. I didn’t plan to release a standalone version anyway so no need to implement MIDI here.

Emscripten & WebAssembly

You may ask why there is a web version now even though I kept writing about Android?

For me, Emscripten is an intermediate step. I didn’t want to deploy to my Android tablet over and over again to test my progress so I thought that Emscripten as a first challenge would be a good candidate to start with.

As I plan to keep most of the code in C++ anyway, both platforms are more similar than expected. Android an Emscripten are both „based“ on Linux (more or less), they have OpenGL ES as possible renderer and as I use CMake as a build system on all platforms differences are very minor during build.

Additionally, I’m not sure if an Android version of K1v is really useful. the role of Heat Synthesizer is different here, as it contains lots of online functionality (preset exchange) and can connect to a VST proxy for proper DAW integration, but as K1v is a VST already that doesn’t make much sense.
Releasing it for the web on the other hand is a nice challenge and furthermore something that didn’t exist before, at least I’m not aware of any existing VSTGUI to Emscripten ports.

Porting VSTGUI to Emscripten

While my K1v code compiled more or less directly, a lot of effort was needed to get VSTGUI up and running.

Everything I write here is based on VSTGUI 4.9, to be more specific, the develop branch at 2020.06.24 commit dc95f6a2. This is the version I used to develop the UI for Nils‘ K1v. I did some bug fixes and fixed Linux compilation errors at this time, but other than that, this is the state I’m currently working with / what the Emscripten version is based on.

The VSTGUI core code (everything that is platform independent) only had smaller compilation problems in one or two cases, but the largest issue was that I didn’t have anything for this new platform: No frame, no window, no run loop etc.

I decided to start with the Linux version. The Linux version is based on Cairo for rendering, uses X11/XCB for input handling and GDK for the main loop in Standalone builds.

Cairo for rendering

I knew that Cairo can be compiled in a way to use OpenGL ES for rendering, which both Android and Emscripten support. So i decided to fork Cairo and compile it for Emscripten using CMake.

This brought a lot of dependencies: zlib, libpng, pixman, libpixman.

More projects that I had to fork. Some of them already had CMake build scripts, which was great, for others, I created CMake build scripts from scratch. It took some time but worked eventually.

Although the web assembly based executable linked fine, I still had a lot of work to do. I didn’t solve the standalone part yet, i.e. the window handling and mouse & keyboard input.

I created some stub classes to be able to test it. At last, I heard the K1v preset IA-1 Ahh playing Note C3 in a browser window, yeah 😎

Using SDL2 to have a render target for Cairo

The next problem I had to solve was to define some sort of „window“ as that is what a standalone app in VSTGUI is based on. The problem is that you do not have any sort of window in Emscripten, as you’re running in a web browser. You get a canvas that you can paint on, that’s it.

Luckily, the Emscripten team already solved this problem by porting the SDL library to Emscripten. This was great news as the SDL library also supports Android.

I spent quite some time connecting an SDL „window“ to Cairo. I wanted to give hardware acceleration to Cairo so I compiled Cairo with OpenGL ES support and created a GL Surface in SDL. Unfortunately, I had to drop this idea for Emscripten as only a subset of OpenGL ES 2.0 is implemented and Cairo makes use of these missing features (for example pbuffers).

So instead of using the GL based backend, Cairo runs in software at the moment. Nevertheless, I was blown away when I saw parts of my UI for the first time:

Font Support

As you might have noticed, there are no texts in the UI. The reason for this is that the Linux build of VSTGUI uses fontconfig to enumerate the available system fonts at startup. I had to comment out that whole code block as there is no fontconfig support in Emscripten and adding this library doesn’t make sense as there are no „system fonts“ available anyway. So instead of using fontconfig, I did the following:

I created a virtual file system for Emscripten at compile time with a folder named „fonts/“ that contains all fonts that I need for K1v. These are not many and the resulting data blob is around 1Mib in size only.
At load time, this font list is built by enumerating through all files in this folder. Then, I load the files with FreeType to query for the family name and style name to finally rebuild the original fonts list that VSTGUI uses. This worked out pretty well and is completely customizable as the available fonts depend on what you put into the virtual filesystem. No hardcoded hacks were necessary.

Input Handling

As I had the SDL library, adding input handling was pretty straightforward. I use the SDL for mouse, keyboard & window events. I also use the SDL for timers.

Missing Features

The project is highly experimental and in a very early stage. VSTGUI support lacks the following:

• file selector is not available
• no alert boxes
• no support window styles as there is no real window
• no drag & drop
• no tooltips
• no clipboard support

The web version of Nils‘ K1v doesn’t currently support:

• State is not preserved, if you reload the page, you’re going to have factory patches again, no matter what you edited before
• MIDI – The only possible input is the virtual keyboard on the web page
• Load or Save of files due to missing VSTGUI file selector

Most of the above could be implemented easily. Another thing worth mentioning is that I don’t currently use AudioWorklets as it seems to be quite cumbersome for now to get them running in Emscripten. The audio output is done by OpenAL and everything is single threaded. You can expect some audio pops & clicks here or there, it depends on your browser & computer, mostly noticeable when large UI parts are redrawn. Comments about how it works for you would be nice.

Future Work

After finishing the port to Emscripten, its time for some cleanup work, obviously. After I will have finished cleaning up the VSTGUI parts, I’m going to make my VSTGUI fork open source. I might decide to do a pull request, maybe Steinberg is interested to integrate official Emscripten support? Who knows.

To summarize, I had to create implementations for the following VSTGUI interfaces to make VSTGUI work with Emscripten:

• VSTGUI::IPlatformTimer – SDL timers
• VSTGUI::IPlatformFrame – SDL <=> Cairo frame
• VSTGUI::Standalone::Platform::IWindow – SDL Window
• For standalone mode, an SDL application which drives VSTGUI::Standalone::Detail::Application
• Some groundwork such as an SDL run loop to drive redraws & inputs.

These are all great preparation steps to have an Android port of Heat Synthesizer. I’m going to continue this journey with the K1v project though, as I want to proof that my concept works before doing a complete UI redesign of Heat Synthesizer with VSTGUI.

I’m confident that the Android version of the SDL/libcairo combination will support hardware accelerated rendering, which is even more important on mobile. As Android has full OpenGL ES 2 support, this should work fine.

I’m going to post updates on this topic. Leave a comment and tell me what you think 👍