A Bit of an Adventure
Finding an AspNetCore Application Bitness
When you want to know how things really work, study them when they’re coming apart.
- William Gibson
As an engineer, there is nothing more exciting than a problem for which there is still no solution. The search to understand how something works is something that is in the nature of engineers and that excites us in a funny way, it’s like a new adventure ready to begin.
This adventure started with some OutOfMemory errors in the recycling of our application under high load, alerted by our monitoring dashboards. This was a strange thing to happen since there wasn’t a major change in our application that could cause this. So we started looking for the root cause of this problem.
Our Application was in AspNetCore 2.2 and was hosted in WindowsServer 2016
Our first thought was to look for a Memory Leak, so we proceeded to extract the dump files from the application in order to look where a memory leak could potentially be.
Nothing, we didn't found any strange behavior in the dump files…
So apparently there wasn’t any Memory Leak in our application, but the truth is that this kind of errors are generally related to memory leaks. What could be causing this?
So we analyzed how much memory was being used by the application and we discovered something that we were not expecting. The application was running at 32-bits!
A 32-bit processor includes a 32-bit register, which can store 232 or 4,294,967,296 values. One bit in the register can reference an individual byte in memory, so a 32-bit system can address a maximum of 4 gigabytes (4,294,967,296 bytes) of RAM.
Our application was supposed to be running under a 64-bit process. Why was this happening?
We talked with other teams that had their applications hosted in different machines than ours and found out that their applications were running in 64-bit processes. We validated the IIS option to Enable 32-bit applications and this option was set up to false.
We still had no explanation for the reason behind our applications were running in 32-bit processes. This is where we looked at the command that was being used to launch our application and this is where things start to get interesting.
After some googling, we found out that the dotnet command is configured as a system environment variable. With this, we went to validate our system variables and we noticed that we had two entries for this variable in the following order:
C:\Program Files (x86)\dotnet
When resolving a PATH variable the OS will use the first match found! This made our applications ran under 32-bit processes!
So we changed the order of the dotnet variables recycled the application and it started in a 64-bit process as we initially expected it was running!
As engineers, this wasn’t enough, we understood the cause for the problem but now we need to standardize the application initialization so that we are not dependent on the order of the path variables to control whether our application will run in 32 or 64 bits. So we looked for the guidelines and best practices application initialization.
After some analysis, we came up with some solutions to avoid this coupling between the order of the PATH variables and the bitness of the application process.
Upgrade to AspNetCore 3.1
The AspNetCore 3.1 is the latest supported version of the dotnet framework. In this version, we don’t have the Web.config file and even if the dotnet 32-bit PATH variable is on top the application will start under a 64-bit process.
Publishing the application using -r win-x64
Publish the application using the following command:
dotnet publish -r win-x64 --self-contained false
Using this option the x64 process bitness will be forced and if the 32-bit dotnet PATH variable is first in the list, an error will be returned and the application won’t start.
Starting the application using the dotnet.exe
Starting the application through the *.exe file would make the application always start under a 64-bit process by default.
Modify the web.config file %LAUNCHER_PATH% Variable
The %LAUNCHER_PATH% can be found in the Web.config file of AspNetCore applications, this variable can be replaced by the x64 dotnet path and the application will start under a 64-bit process.
Wrapping things up
Any of the above solutions will ensure that your application starts under a 64-bit process without relying upon the order of your dotnet PATH variables.
Finding the solution to this problem was a real adventure. The passion for resolving enigmatic problems is the heart of engineering and that makes me love my job.