@auduchinok You've mentioned that you have seen some slowness too, right?

We are seeing this one right now.

So, judging by the tracing, it is doing something in background (i.e. not completely stuck), but is very slow.

@auduchinok You've mentioned that you have seen some slowness too, right?

Yes, it looked like the same place in the tables generation. I've used build.cmd -noVisualStudio on a clean local copy.

So, when building with .NET7-p7:

And when with .NET7-p2 (dev17.3 branch for example):

Spends almost 3 times less building compiler.

Will try and collect snapshot diffs.

Scale is different, but it's clear that it goes from the same stages in beginning (parsing probably?) and the end (emitting assembly?).

Some experimenting:
Same fsharp compiler repo commit (HEAD).
Build with preview2 of SDK - proto build is ok ~90s
Build with preview3 of SDK - proto build is very slow (8m and counting)

From discussion with @Maoni0 and @mangod9

• The problem is related to .NET 7 enabling GC regions by default, particularly for "workstation GC". For example the initial compilation of the F# compiler:

  With .NET 6.0:

  > dotnet.exe C:\Program Files\dotnet\sdk\6.0.400\FSharp\fsc.dll"  @..\..\..\log.txt
  TIME:  0.4 Delta:  0.3 Mem:  98 G0:   6 G1:  3 G2:  2 [Import mscorlib and FSharp.Core.dll]
  TIME:  9.5 Delta:  9.1 Mem: 339 G0:  65 G1: 24 G2:  4 [Parse inputs]
  TIME:  9.5 Delta:  0.0 Mem: 339 G0:   0 G1:  0 G2:  0 [Import non-system references]
  TIME: 32.2 Delta: 22.7 Mem: 746 G0: 1063 G1: 163 G2:  1 [Typecheck]
   ...
  

  With .NET 7.0 Preview 7 on exactly same inputs

  > dotnet.exe "C:\Program Files\dotnet\sdk\7.0.100-preview.7.22377.5\FSharp\fsc.dll"  @..\..\..\log.txt
  TIME:  0.3 Delta:  0.2 Mem: 104 G0:   6 G1:  4 G2:  2 [Import mscorlib and FSharp.Core.dll]
  TIME:  8.3 Delta:  8.0 Mem: 373 G0:  65 G1: 31 G2: 16 [Parse inputs]
  TIME:  8.3 Delta:  0.0 Mem: 373 G0:   0 G1:  0 G2:  0 [Import non-system references]
  TIME: 223.4 Delta: 215.0 Mem: 832 G0: 1063 G1: 651 G2: 616 [Typecheck]
  

  Note the number of Gen2 collections in this phase of the compilation has gone up from 1 to 616.

• The very large slowdown disappears if we turn on ServerGC for fsc - this can be done globally via an environment variable or we can add it as a runtimeconfig.json for the F# compiler in the .NET SDK. However this is problematic as ServerGC is about 30% slower for the F# compiler.

Repro.

1. Get this repo and checkout the current main:

   git clone https://github.com/dotnet/fsharp
   git checkout 75e1f5f1900a351ca036081cf2d1ed453ddf5ef1
   

2. .\build -noVisualStudio

   NOTE: this unfortunately takes about 15 minutes because the slowdown in the SDK compiler, but it gets all the right bits in place for the next step.

3.  cd src\Compiler
   

4. Copy https://gist.github.com/dsyme/07499d160e3d16e60aa2af03742ba63e to args.txt and adjust paths to nugets (sorry)

5. Do the compilation with whichever F# compiler and .NET runtime you like, e.g.

    dotnet "C:\Program Files\dotnet\sdk\7.0.100-preview.7.22377.5\FSharp\fsc.dll"  @args.txt
   

    dotnet "C:\Program Files\dotnet\sdk\6.0.400\FSharp\fsc.dll"  @args.txt
   

   Note these run on .NET 7 and .NET 6 respectively. You can Ctrl-C after the "Typecheck" phase is reported, no need to wait

You can modify the settings these compilers are using:

To turn off GC regions for .NET 7

set COMPLUS_GCName=clrgc.dll

To turn on Server GC

Edit the appropriate fsc.runtimeconfig.json e.g.

 {
   "runtimeOptions": {
     ...
     "configProperties": {
       ...
       "System.GC.Server": true
     }
   }
 }

I guess as a temporary resolution, we should switch to old GC (via the runtimeconfig, if possible).

Fascinating, I saw this happen yesterday too when building the compiler with my own built runtime to get SuperPMI collections.

I don't know if this is relevant, but main dominators shown in dotmemory are FSharpLists (they behave like linked lists).

My theory is that gen0 budget gets exhausted very quickly and afterwards GC basically keeps scanning all the memory because of huge object graphs. Are regions linked list friendly by the way?

Traces show that most GC events are AllocSmall - Gen2 and very rarely AllocSmall - Gen0

I also was curious about working set - workstation GC was really trying to keep it 600megs max on 64 gb ram machine.

GC time with GCServer went from 90% to just 8%. But this came with working set increase by 10 times (6 gigs).

Also, there were only a few GC collections. I guess this is because there was still much memory left and server GC is not so shy about using it.

@En3Tho I'd advise against analysing too much at this stage. Whatever else, the GC was handling this application (the F# compiler) beautifully before and it seems to me that simply too much has changed in the GC and much more caution is now needed - in particular there has to be an easy fallback to get the old characteristics, and I'd also question whether region collecting should be on by default. In any case methodologically we just need to focus here on finding a way back to the rock-solid reliable performance we've had for the last 15 years.

One relevant factor is that the application turns on Batch mode.

@vzarytovskii @KevinRansom You might want to look at turning on ServerGC (or undoing the update to .NET 7 preview 7 back to preview 2) so that we are able to work in this repo

we should switch to old GC (via the runtimeconfig, if possible).

@mangod9 said this isn't possible via runtimeconfig in preview7.

Turning server mode on should be possible

• For the SDK compiler in the existing SDKs, an environment variable export DOTNET_gcServer=1 set in the build should do the trick
• For the Bootstrap compiler, on Windows it runs using .NET Framework - on Mac/Linux we should generate a runtimeconfig
• For the output compiler to be inserted into future SDKs we should use a runtime config. There's an existing runtimeconfig in the SDK, I'm not sure where/how that gets created or how to adjust it. THis is what's currently produced in main:

C:\Windows\System32>more \GitHub\dsyme\fsharp\artifacts\bin\fsc\Debug\net6.0\fsc.runtimeconfig.json
{
  "runtimeOptions": {
    "tfm": "net6.0",
    "framework": {
      "name": "Microsoft.NETCore.App",
      "version": "6.0.0"
    },
    "rollForwardOnNoCandidateFx": 2
  }
}

This has been fixed by .NET team