Adventures in Cygwin’s fork performance

Introduction to the problem

As you may know, I maintain a few packages for Cygwin. Cygwin is a compatibility layer for Windows that makes it easier to run applications designed to be run on Linux (details at www.cygwin.com).

Due to limitations of Windows and the NT kernel, Cygwin is often a lot slower than Linux when running, and one particular slowdown is due to the complex way Cygwin has to emulate the Linux fork() call. This function is used when creating processes, so things like bash scripts and make files tend to run very slowly under Cygwin.

I decided to investigate my issues because my package builds were taking an extortionate amount of time, even when compared to other much slower systems other Cygwin maintainers were using.

I’ve done a few other posts on Cygwin, such as this old one. DDRescue-GUI also uses Cygwin in order to support Windows.

DISCLAIMER: This is an informational guide – I won’t take responsibility for you damaging your Windows installation. Please note that most of the changes I made here had a minimal, hard to measure impact, but I believe that’s because QEMU/KVM already eliminated most of the overhead. I suspect this guide will be very useful to those of you running VirtualBox, but cannot confirm this. The general troubleshooting steps should be useful.

NOTE: Something I forgot to mention is that the decline in fork performance over time was eliminated by these changes – it started off “fast” in relative terms, but then became so poor it was becoming a significant time sink when building packages. Additionally, my package build time was dramatically reduced after making these changes. Unfortunately, I didn’t measure the before and after so am unable to demonstrate the change.

UPDATE: There is a general guide to help you get started with Cygwin at https://www.pcwdld.com/cygwin-cheat-sheet.

Setup Details

I was originally running Windows inside VirtualBox, which it turns out is a LOT slower than using QEMU/KVM with virt-manager. A lot of the performance issues were fixed just by using QEMU/KVM, but obviously this will only work on Linux hosts. The fork speed seems to drop off with repeated attempts in Cygwin, but not in Linux.

Here are the details of my setup:

CPU: 6-core Ryzen 3600 (at stock speeds).

RAM: 32 GB (16 GB dedicated to Windows VM)

Hypervisor: QEMU/KVM (hardware acceleration on).

Storage: 100 GB virtual disk connected with VirtIO (much quicker than emulating SATA).

Antivirus: Avira Free Antivirus (with exceptions defined for Cygwin).

Windows version: Windows 10 20H2 (though this walkthrough is probably also relevant to earlier versions of Windows). Almost a clean install, minimal extra applications added.

Host OS: Linux Mint 20 Cinnamon (64-bit).

Cygwin: 3.1.6, 64-bit (32-bit Cygwin is a bit unreliable and erratic due to some limitations). CYGWIN environment variable is set to “winsymlinks:lnk proc_retry:3”.

The video

The video demonstration.

Initial Benchmarks

In order to measure the performance difference, I’m using a fork benchmark program, available at https://github.com/mondalaci/fork-benchmark. I ran the benchmark 3 times for each run and average the results, with 100 forks each time, and clean boots between attempts.

I decided to benchmark my Linux host OS as well, just for interest and to demonstrate how much slower Cygwin is compared to Linux, while making sure to close as many programs as possible. Note that my screen recorder will have slowed things down a bit.

Linux: 0.23 seconds (average of 3 runs).

Cygwin: 13 seconds (to the nearest second, with drop-off over time).

Diagnosis steps

Disabling my antivirus (sanity check)

I had already made exceptions for the Cygwin folders, but I thought it would be a good idea to double-check that my antivirus wasn’t somehow interfering. I suggest you pick a “disable until restart” option or similar, just in case you forget to re-enable your antivirus.

Results: 13 seconds (to the nearest second, with drop-off).

This was good, as it meant I had configured my antivirus correctly. If Cygwin is suddenly much faster, you may need to check that you have defined exceptions for the C:\Cygwin folder (or wherever else you installed Cygwin).

Debloat script

This script disables a lot of Windows 10 bloatware that slows down your system, courtesy of Chris Titus Tech on YouTube. My systems run noticeably faster for day to day use after running this. Skip this if you aren’t running Windows 10.

The instructions for this are available at https://christitus.com/debloat-windows-10-2020/. I picked the “Essential Tweaks”, then I disabled Cortana and OneDrive, then “Background Apps”, which limits the number of apps running in the background.

Please note: It is a good idea to backup your data first before doing this.

Results: 12.2 seconds (with drop-off). A small improvement, but probably within the margin of error.

Changing Windows virtual memory settings (VM only)

I disabled the paging file in Control Panel > System > Advanced System Settings. You probably don’t want to do this on bare metal, but it may improve VM performance.

Results: 13.2 seconds (with drop-off). Probably within the margin of error.

Change Windows power management settings

You may not want to do this on bare metal either, to save power.

Windows defaults to the balanced plan, but there is also a “Performance” plan, which may improve performance slightly. I also went through the individual settings but didn’t change anything much.

Results: 12.6 seconds (with drop-off). A small improvement, but probably within the margin of error. We did get the quickest ever individual time of 10.4 seconds though.

Change Windows performance settings

Windows has a bunch of general performance and graphical fidelity settings that may reduce performance (especially when, like me, you have no video acceleration). This is also found in Advanced System Settings.

I disabled everything except font smoothing, window content while dragging, and window shadows. You may also want desktop icon shadows.

Results: 12.8 seconds (with drop-off). Probably within the margin of error. Again. I’m getting a bit frustrated at this point!

Disabling System Restore

Again, you probably only want to do this if you’re using a VM and have snapshots/backups. This option can be found in System Properties > System Restore. You can also delete old restore points here.

Results: 12.2 seconds (with drop-off). A small improvement, but probably within the margin of error.

Edit cygserver (Cygwin Server) options

This is a Cygwin-specific configuration change. Details of what could be changed are at https://cygwin.com/pipermail/cygwin-apps/2020-December/040796.html (click “previous message by thread” a few times for more context).

Results: 12.2 seconds (with drop-off). Exactly the same!

Disable some Windows services

Do this at your own risk. This may seriously impact the stability or reliability of your system. I used this article as a guide: https://www.thewindowsclub.com/which-windows-10-services-safe-to-disable.

You could also disable unnecessary startup programs from “msconfig” or the Task Manager.

Please watch the video on YouTube if you want to see exactly what I changed.

Results: 12.6 seconds (with drop-off).

Extra: KVM CPU configuration

By default, KVM specifies lots of CPU sockets, rather than cores or threads. Windows doesn’t support more than 2 CPUs, so you may need to change this to improve performance. I can’t believe I missed this.

Results: 12.6 seconds (with drop-off). This is disappointing, but I guess this is still within the margin of error.

Cygwin Performance Conclusion

Sometimes it’s better to start with the simple things! I do find it interesting that forking was quicker with 2 cores than it was with 6 in VirtualBox, though. I completed a package build after the end of the video, and it was significantly quicker than before, which was nice.

Hopefully this has been useful to you, and see you in the next post 🙂

Tagged , , . Bookmark the permalink.

About Hamish McIntyre-Bhatty

I'm a self-employed software developer working on Free Software projects, as well as studying for my degree with the Open University. Being pedantic when it comes to detail is fortunately useful for both of these things! A strong believer in free software, I have a few pay-for programs available under the GPLv3 and enjoy reporting bugs and helping to improve various open source projects, including volunteering at Wimborne Model Town to work on their river control system.

5 Responses to Adventures in Cygwin’s fork performance

  1. Adelaide says:

    So, essentially, an hour long video, a long article, and a misleading title, all to say that no Cygwin or Windows 10 settings will improve fork performance.

    • I see your point, I will change the title to be clearer.

      To be fair, there is a disclaimer at the top of the article, and it is noted that I saw a significant improvement when building packages at the bottom if the article, which suggests my testing method wasn’t showing many of the improvements. I couldn’t show the improvement because sadly I neglected to record the time properly for that package build before starting work on this post/video.

      As I said though, I will change the title and add the disclaimer to the video description. Even as it is, it is hopefully helpful to someone who is having really poor performance.

  2. Johnny Norre says:

    Have you seen this video? https://www.youtube.com/watch?v=qbKGw8MQ0i8 regarding rust is slow to install on Windows 10 (and a possible solution)

  3. This video by Chris Titus Tech: https://www.youtube.com/watch?v=Kq849CpGd88 illustrates the performance difference I was seeing after switching from Virtualbox to KVM/QEMU.

    I really wish I had taken some benchmarks before the switch, but sadly I didn’t, and it would be a lot of effort to spin up a new Windows VM in Virtualbox just to do this.

Leave a Reply

Your email address will not be published. Required fields are marked *