Jekyll2021-12-09T18:08:45-08:00https://kirija.github.io/feed.xmlMarc-Olivier KillijianProfessional Website{"name"=>nil, "avatar"=>"MOK.jpg", "bio"=>"Directeur de Recherche CNRS on secondment, Professor of Computer Science UQAM", "location"=>"UQAM, MTL", "email"=>"killijian.marc-olivier.2@uqam.ca", "office"=>"Local PK-4740", "googleMaps"=>"https://goo.gl/maps/uuGS91n54YS2", "googlescholar"=>"https://scholar.google.com/citations?hl=fr&user=SGPYrpA69BsJ", "github"=>"kirija", "twitter"=>"kirija", "linkedin"=>"marcokillijian", "researchGroup"=>"SecPriv@UQAM", "researchUri"=>"", "dblp"=>"https://dblp.uni-trier.de/pers/hd/k/Killijian:Marc=Olivier.html"}killijian.marc-olivier.2@uqam.caCompiling MESONH on Apple Silicon 2 : arm642021-07-03T00:00:00-07:002021-07-03T00:00:00-07:00https://kirija.github.io/blog-post-1<p>Recently, I’ve been trying to help someone to install the <a href="http://mesonh.aero.obs-mip.fr/">Meso-NH atmospheric model</a> on a new Macbook Air M1. In this post, I try to document how I partially succeeded. The complete x86_64 story has been documented earlier, here I’m interested with the native arm64 version.</p>
<p>In three words : easy to compile, there’s a strange bug during the execution of any of the produced binaries (Illegal instruction) somewhere in hwloc that might be due to some mis-alignment. I did not resolve this issue yet but found a way to circumvent it by using <em>mpirun</em>, which enables to run Meso-NH natively on the M1.</p>
<h1 id="compiling-meso-nh-for-arm64">Compiling Meso-NH for arm64</h1>
<p>Easy peasy, just install homebrew, git and open-mpi. Make sure to make the symbolic links to gcc, g++ and gfortran if needed, as described in the x86_64 post. Configure, make, that’s it !</p>
<h1 id="running-meso-nh-natively-on-bigsur-on-a-m1">Running Meso-NH natively on BigSur on a M1</h1>
<p>As mentioned earlier, there’s an issue in the device discovery launched by hwloc which raises a SIGILL, here’s my backtrace</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ src git:(MNH-54-branch) ✗ lldb ./dir_obj-LXgfortran-R8I4-MNH-V5-4-4-MPIAUTO-DEBUG/MASTER/DIAG
(lldb) target create "./dir_obj-LXgfortran-R8I4-MNH-V5-4-4-MPIAUTO-DEBUG/MASTER/DIAG"
r
bt
Current executable set to '/Users/UQAM/Code/MESONH/MNH-V5-4-2/src/dir_obj-LXgfortran-R8I4-MNH-V5-4-4-MPIAUTO-DEBUG/MASTER/DIAG' (arm64).
(lldb) r
Process 68887 launched: '/Users/UQAM/Code/MESONH/MNH-V5-4-2/src/dir_obj-LXgfortran-R8I4-MNH-V5-4-4-MPIAUTO-DEBUG/MASTER/DIAG' (arm64)
bt
2021-07-03 22:16:43.524938-0400 orted[68890:6584537] +[MTLIOAccelDevice registerDevices]: Zero Metal services found
2021-07-03 22:16:43.525970-0400 orted[68890:6584537] [CL_INVALID_OPERATION] : OpenCL Error : Failed to retrieve device information! Invalid enumerated value!
2021-07-03 22:16:43.525981-0400 orted[68890:6584537] [CL_INVALID_OPERATION] : OpenCL Error : Failed to retrieve device information! Invalid enumerated value!
2021-07-03 22:16:43.525986-0400 orted[68890:6584537] [CL_INVALID_OPERATION] : OpenCL Error : Failed to retrieve device information! Invalid enumerated value!
Process 68887 stopped
* thread #1, queue = 'com.Metal.DeviceDispatch', stop reason = EXC_BAD_INSTRUCTION (code=1, subcode=0x1e220820)
frame #0: 0x0000000117b38e94 AGXMetal13_3`___lldb_unnamed_symbol3036$$AGXMetal13_3 + 160
AGXMetal13_3`___lldb_unnamed_symbol3036$$AGXMetal13_3:
-> 0x117b38e94 <+160>: fmul s0, s1, s2
0x117b38e98 <+164>: fcmp s0, #0.0
0x117b38e9c <+168>: mov w13, #0x44600000
0x117b38ea0 <+172>: fmov s1, w13
Target 0: (DIAG) stopped.
(lldb) bt
* thread #1, queue = 'com.Metal.DeviceDispatch', stop reason = EXC_BAD_INSTRUCTION (code=1, subcode=0x1e220820)
* frame #0: 0x0000000117b38e94 AGXMetal13_3`___lldb_unnamed_symbol3036$$AGXMetal13_3 + 160
frame #1: 0x0000000117967754 AGXMetal13_3`___lldb_unnamed_symbol484$$AGXMetal13_3 + 1796
frame #2: 0x000000011796ae38 AGXMetal13_3`___lldb_unnamed_symbol507$$AGXMetal13_3 + 52
frame #3: 0x0000000192b7cf1c Metal`-[MTLIOAccelService initWithAcceleratorPort:] + 400
frame #4: 0x0000000192b7cd54 Metal`+[MTLIOAccelService registerService:] + 160
frame #5: 0x000000018ad43ec0 libdispatch.dylib`_dispatch_client_callout + 20
frame #6: 0x000000018ad52fb4 libdispatch.dylib`_dispatch_lane_barrier_sync_invoke_and_complete + 60
frame #7: 0x0000000192c2cff0 Metal`MTLIOAccelServiceRegisterService + 108
frame #8: 0x0000000192b7cb4c Metal`+[MTLIOAccelDevice registerDevices] + 264
frame #9: 0x0000000192ba53d0 Metal`invocation function for block in MTLDeviceArrayInitialize() + 1284
frame #10: 0x000000018ad43ec0 libdispatch.dylib`_dispatch_client_callout + 20
frame #11: 0x000000018ad45758 libdispatch.dylib`_dispatch_once_callout + 32
frame #12: 0x0000000192b7c904 Metal`MTLCopyAllDevices + 244
frame #13: 0x00000001178b6598 AppleMetalOpenGLRenderer`GLDDeviceRec::initWithDisplayMask(unsigned int) + 140
frame #14: 0x00000001178bc734 AppleMetalOpenGLRenderer`gldCreateDevice + 216
frame #15: 0x00000001ccefd1c4 libGFXShared.dylib`gfxInitializeLibrary + 1920
frame #16: 0x00000001cd2adce0 OpenCL`___lldb_unnamed_symbol731$$OpenCL + 432
frame #17: 0x000000018aeef608 libsystem_pthread.dylib`__pthread_once_handler + 80
frame #18: 0x000000018af3c04c libsystem_platform.dylib`_os_once_callout + 32
frame #19: 0x000000018aeef59c libsystem_pthread.dylib`pthread_once + 100
frame #20: 0x00000001cd2adab4 OpenCL`___lldb_unnamed_symbol728$$OpenCL + 116
frame #21: 0x00000001cd27f8ec OpenCL`clGetDeviceIDs + 168
frame #22: 0x0000000110202548 libhwloc.0.dylib`hwloc_opencl_discover(backend=<unavailable>, dstatus=<unavailable>) at topology-opencl.c:86:13
frame #23: 0x00000001101da5d4 libhwloc.0.dylib`hwloc_discover_by_phase(dstatus=0x000000016fdfd510, phasename=<unavailable>, topology=<unavailable>, topology=<unavailable>) at topology.c:3290:5
frame #24: 0x00000001101e1fc8 libhwloc.0.dylib`hwloc_topology_load(topology=0x0000000110a18730) at topology.c:3493:5
frame #25: 0x00000001107154dc libopen-pal.40.dylib`opal_hwloc_base_get_topology + 3756
frame #26: 0x000000011064f520 libopen-rte.40.dylib`orte_ess_base_proc_binding + 3484
frame #27: 0x0000000110d0705c mca_ess_singleton.so`rte_init + 5192
frame #28: 0x00000001106814c8 libopen-rte.40.dylib`orte_init + 676
frame #29: 0x00000001103e7264 libmpi.40.dylib`ompi_mpi_init + 912
frame #30: 0x0000000110398fc4 libmpi.40.dylib`PMPI_Init + 120
frame #31: 0x00000001102bdf3c libmpi_mpifh.40.dylib`mpi_init_ + 40
frame #32: 0x0000000106106b0c DIAG`__mode_mnh_world_MOD_init_nmnh_comm_world at spll_mode_mnh_world.f90:46:30
frame #33: 0x000000010568460c DIAG`__mode_io_ll_MOD_initio_ll at spll_mode_io_ll.f90:90:35
frame #34: 0x0000000100002d50 DIAG`MAIN__ at spll_diag.f90:246:16
frame #35: 0x000000010000cdb8 DIAG`main at spll_diag.f90:94:4
frame #36: 0x000000018af11450 libdyld.dylib`start + 4
(lldb)
</code></pre></div></div>
<p>I spent some time in order to see what was going on but was’t able, yet, to understand the problem. However, I discovered that if you launch the same binary using mpirun, the calls to hwloc are not exactly similar and circumvent the problem, making it possible to run correctly !</p>
<h2 id="thanks">Thanks</h2>
<p>I’d like to thank <a href="https://www.coudert.name">FXCoudert</a> which helped me prune some branches related to gfortran and the M1 ecosystem.</p>{"name"=>nil, "avatar"=>"MOK.jpg", "bio"=>"Directeur de Recherche CNRS on secondment, Professor of Computer Science UQAM", "location"=>"UQAM, MTL", "email"=>"killijian.marc-olivier.2@uqam.ca", "office"=>"Local PK-4740", "googleMaps"=>"https://goo.gl/maps/uuGS91n54YS2", "googlescholar"=>"https://scholar.google.com/citations?hl=fr&user=SGPYrpA69BsJ", "github"=>"kirija", "twitter"=>"kirija", "linkedin"=>"marcokillijian", "researchGroup"=>"SecPriv@UQAM", "researchUri"=>"", "dblp"=>"https://dblp.uni-trier.de/pers/hd/k/Killijian:Marc=Olivier.html"}killijian.marc-olivier.2@uqam.caRecently, I’ve been trying to help someone to install the Meso-NH atmospheric model on a new Macbook Air M1. In this post, I try to document how I partially succeeded. The complete x86_64 story has been documented earlier, here I’m interested with the native arm64 version.Compiling MESONH on Apple Silicon 1 : x86_642021-07-01T00:00:00-07:002021-07-01T00:00:00-07:00https://kirija.github.io/blog-post-1<p>Recently, I’ve been trying to help someone to install the <a href="http://mesonh.aero.obs-mip.fr/">Meso-NH atmospheric model</a> on a new Macbook Air M1. In this post, I try to document how I partially succeeded.
In three words : up to now I have been able to compile it for x86_64 (on the arm64 based M1) and run it successfully using Rosetta2 on the M1, but I’m still stuck at the linkage/run part of a native arm64 version, I’ll document that later.</p>
<h1 id="compiling-meso-nh-for-x86_64-emulation">Compiling Meso-NH for x86_64 emulation</h1>
<p>The arm64 ecosystem is still in its infancy, not all compiler vendors, virtual machine managers, homebrew packets, have yet a version functioning natively on Apple Silicon. Fortunately, the Apple Rosetta2 system, which emulates x86_64 on top of arm64, works pretty well. This is what we’ll be using here to compile and run Meso-NH.</p>
<h2 id="x86_64-and-arm64-modes">x86_64 and arm64 modes</h2>
<p>In a nutshell, if you want to force the system to execute something in a mode, you can use the <em>arch</em> command :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>arch -arm64 _your_command_to_be_executed_in_arm64_mode_
arch -x86_64_ _your_command_to_be_executed_in_x86_64_mode_
</code></pre></div></div>
<p>Another interesting command I used intensively during my study was the <em>lipo</em> command that you can use to check (and also create) architecture information of a binary file :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lipo -info _your_exe_
</code></pre></div></div>
<p>This will tell you wether or not the binary is fat (contains multiple architecture executables) and for which architectures it is.</p>
<h2 id="installing-homebrew-for-x86_64">Installing Homebrew for x86_64</h2>
<p>At first, let’s consider that you have a fresh M1 install and not yet installed homebrew on your machine. If you had, we’ll discuss your case later, but keep reading this part.</p>
<p>In order to compile Meso-NH for x86<em>64, we need to get x86_64 version of _open-mpi</em>, <em>gcc</em> and <em>git</em> and for that, we are going to install a x86_64 homebrew.
That’s going to be relatively simple if you have never installed homebrew on this machine (or didn’t migrate an old homebrew from your previous Mac), otherwise i recommend some cleanup (if you need to, list the packages you had installed) :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/uninstall.sh)"
</code></pre></div></div>
<p>and if that’s needed, even a force clean</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo rm -fr /usr/local
</code></pre></div></div>
<p>Now we can go with the fresh install</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
</code></pre></div></div>
<p>And add homebrew binaries to your <em>path</em> if it’s not already the case :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export PATH="/usr/local/bin:$PATH"
</code></pre></div></div>
<p>That’s it, you’ve got a fresh and clean install of a x86_64 homebrew installed and can start installing the compilation chain and dependancies, but to make sure that you’ll be running that x86_64 version, you can check that :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>which brew
</code></pre></div></div>
<p>points to <em>/usr/local/bin/brew</em></p>
<h2 id="installing-x86_64-dependancies">Installing x86_64 dependancies</h2>
<p>Installing <em>open_mpi</em> will imply installing <em>gcc</em>, we also need <em>git</em> :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>brew install git open-mpi
</code></pre></div></div>
<p>Now, we need to create symlinks to gcc and g++ (i don’t know why but gfortran is already linked in gcc-11 and was not in gcc-5!) :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd /usr/local/bin
rm gcc g++
ln -s gcc-11 gcc
ln -s g++-11 g++
</code></pre></div></div>
<p>A safety check that you’re running gcc v11 :
gcc –version</p>
<h2 id="compiling-x86_64-meso-nh">Compiling x86_64 Meso-NH</h2>
<p>Now, you should be in a controlled territory and your usual Meso-NH compilation should work :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd src
./configure
source ../conf/_your_config_file_
make -j12
</code></pre></div></div>
<h2 id="what-if-you-already-had-an-arm64-homebrew-">What if you already had an arm64 homebrew ?</h2>
<p>Well, that’s not too bad, you can easily deal with a dual homebrew and have packages installed for different, or both, architectures.</p>
<p>First you can optionally alias brew in order to easily choose which one you are dealing with :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>alias ibrew='arch -x86_64 /usr/local/bin/brew'
alias abrew='arch -arm64 /opt/homebrew/bin/brew'
</code></pre></div></div>
<p>Then, I’d recommend launching a shell using the architecture you are going to use, and setup the paths according to the architecture you’re running :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>arch -x86_64 zsh
arch -arm64 zsh
</code></pre></div></div>
<p>And add those lines to your <em>.zshrc</em> :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>architecture=`arch`
if test "${architecture}" = "i386"; then
echo "Using Rosetta2 for x86 emulation"
export PATH="/usr/local/bin:$PATH"
elif test "${architecture}" = "arm64"; then
echo "Benefitting from the Apple Silicon greatness"
export PATH="/opt/homebrew/bin:$PATH"
else
echo "Hello Houston ? We have a problem here."
fi
</code></pre></div></div>
<h2 id="sources">Sources</h2>
<ul>
<li>(Hi I’m Sam)[https://soffes.blog/homebrew-on-apple-silicon]</li>
<li>(Homebrew GitHub discussions)[https://github.com/Homebrew/discussions/discussions/149]</li>
</ul>{"name"=>nil, "avatar"=>"MOK.jpg", "bio"=>"Directeur de Recherche CNRS on secondment, Professor of Computer Science UQAM", "location"=>"UQAM, MTL", "email"=>"killijian.marc-olivier.2@uqam.ca", "office"=>"Local PK-4740", "googleMaps"=>"https://goo.gl/maps/uuGS91n54YS2", "googlescholar"=>"https://scholar.google.com/citations?hl=fr&user=SGPYrpA69BsJ", "github"=>"kirija", "twitter"=>"kirija", "linkedin"=>"marcokillijian", "researchGroup"=>"SecPriv@UQAM", "researchUri"=>"", "dblp"=>"https://dblp.uni-trier.de/pers/hd/k/Killijian:Marc=Olivier.html"}killijian.marc-olivier.2@uqam.caRecently, I’ve been trying to help someone to install the Meso-NH atmospheric model on a new Macbook Air M1. In this post, I try to document how I partially succeeded. In three words : up to now I have been able to compile it for x86_64 (on the arm64 based M1) and run it successfully using Rosetta2 on the M1, but I’m still stuck at the linkage/run part of a native arm64 version, I’ll document that later.