The Mac/Linux Port Blog/Thread

[flibitijibibo]

Hello everybody!

For those who haven't heard yet, Waveform is going to be ported to Mac and Linux. I'm the guy that's doing the ports! I'll also be doing some audio programming for Waveform's upcoming expansion, but the ports are my top priority.

I decided to make a blog-ish-type thread so I could keep in touch with those who are eagerly awaiting these ports. I like to keep my development process pretty transparent, as that's what I like to see from other developers. Feel free to ask questions or make suggestions! (Or both!)

June 30, 2012
June 7, 2012
May 21, 2012
May 8, 2012

April 20, 2012

It begins! Kind of!

I've had the source for the last week or so, but I've only had time to read over it and get an idea of what I'll really be doing to get the work done. The reason? Currently finishing up the last two weeks of the semester (finals week: one exam on Monday morning, one exam on Friday evening. College!). After that it should be nothing but Waveform for a good few months, so I apologize for the initial delay.

I'll be digging into it for real next week, but before that, here's a question for you!

Mac users! What type of Mac do you have, and what version of OSX does it have? For example (hypothetical specs below):

2012 Macbook Pro
Mac OSX 10.7 (Lion)
Intel Core i7, 2.2GHz
4GB RAM
AMD Radeon HD 6750M

If you don't know the information, you should be able to get it if you mash the 'More Info' button.

Linux users! A friend of mine, meklu, made a nice sysinfo script that you can download here. Simply run it in terminal and post the output here. It should look like this:

Host:   flibitTower
Kernel: Linux 3.3.1-5.fc16.x86_64 x86_64
Distro: Fedora release 16 (Verne)
CPU:    Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
        8 cores/threads
        3401.00 MHz
RAM:    7.71 GiB
GPU:    GeForce GTX 560 Ti/PCIe/SSE2
        OpenGL 4.2.0 NVIDIA 295.40
        GLSL 4.20 NVIDIA via Cg compiler

I expect the system requirements to be nearly identical to the Windows specs (see bottom), so hardware is probably not a concern. However, I would like to fix any bizarre software incompatibilities before release, so I just need to know what operating systems I need to be testing the most throughout the development process to ensure as few '1.0' problems as possible. I promise to try as many platforms as possible, but I am only one person with very few machines available (luckily I'm within walking distance of two very large Mac labs, so I should be able to test a good range of Mac hardware over there). If the range of platforms is too great I may ask for some testers closer to release, but I'll have to clear that through Eden Industries first.

As for the release itself, it should be pretty straightforward. Mac users can expect a Steamplay release, and I'm hoping that I can bundle the Linbins in a tarball and stick it in the Windows Steam distribution, similarly to Steel Storm and Dungeons of Dredmor. The Linux version obviously won't have Steamworks integration, at least until Valve makes a native Steam client (which they're supposedly working on now (for real this time!)). You'd have to ask Eden about other distribution channels, but that's what my plan is for the Steam version.

Once again, if you have any questions/comments, don't hesitate to reply!

[meklu]

This is quite an interesting endeavour and I should probably start scraping together some money to throw at you people

And here we go with the specs:

Host:   home.meklu.org
Kernel: Linux 3.2.9-gentoo x86_64
Distro: Gentoo Base System release 2.1
CPU:    AMD Phenom(tm) II X4 B25 Processor
        4 cores/threads
        2700.00 MHz
RAM:    11.73 GiB
GPU:    ATI Radeon HD 5700 Series
        OpenGL 4.2.11566 Compatibility Profile Context
        GLSL 4.20

[snuwoods]

okay, here's my specs:

Host: oaksey
Kernel: Linux 3.3.2-1-ARCH x86_64
Distro: Arch Linux
CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
4 cores/threads
2925.24 MHz
RAM: 7.80 GiB
GPU: GeForce 8800 GT/PCIe/SSE2
OpenGL 3.3.0 NVIDIA 295.40
GLSL 3.30 NVIDIA via Cg compiler

oh and FYI, the script required some packages I didn't have installed: mesa-demos, and lsb-release.

[meklu]

Originally Posted by snuwoods

okay, here's my specs:


oh and FYI, the script required some packages I didn't have installed: mesa-demos, and lsb-release.

The Mesa dependency is pretty unavoidable as far as I know. I added some fallbacks for distro detection though. (if flibit hasn't updated it yet, it's here) Did it fail gracefully though?

[zratchet]

2011 27" iMac
Mac OSX 10.6 (Snow Leopard)
Intel Core i5 Quad, 3.1GHz
4GB RAM
AMD Radeon HD 6970M

Mac user here.

[Cheeseness]

Here you go, flibit:

Host:   drenosh
Kernel: Linux 3.3.0-4.fc16.i686.PAE i686
Distro: Fedora release 16 (Verne)
CPU:    Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz
        2 cores/threads
        2999.99 MHz
RAM:    3.94 GiB
GPU:    GeForce GTX 460/PCI/SSE2
        OpenGL 4.2.0 NVIDIA 290.10
        GLSL 4.20 NVIDIA via Cg compiler

Meklu, you have a lot of rams there.

[flibitijibibo]

Well, that was a rather quiet start, wasn't it? Hopefully there won't be this much silence between updates from now on.

I finally started working on the ports "full-time" on April 30. So far all appears to be going well! I've gotten most of the silly work out of the way and I just started getting into the dirtier work, which is where the fun really starts.

One thing I wanted to accomplish with this thread was show anyone curious how porting tends to go, at least from the perspective of someone relatively new to porting commercial games. I imagine someone like icculus has a process that's different from this, but you could say that about any aspect of game development.

The best way to start this blog for real is to go over my current milestone list...

1. Create Unix Makefile based on VCProject - Done!
2. Fix the basic C/C++ things that g++ doesn't like - Done!
3. Port Win32 threading to pthreads - In progress
4. Cross-platform display management and input
5. Port AVI playback to a cross-platform solution (TheoraPlay?)
6. OSX/Linux crash reporting (stack walker, e-mail reports, etc.)
7. Debug/Test the crap out of everything after it finally compiles

So, what on Earth does any of this mean? Let's start from the top...

1. Create Unix Makefile based on VCProject

This generally does not take a whole lot of time, as it's simply a matter of finding the following things:
- Files to compile
- Libraries to include
- Defines to... define
All of this is easily found in a Visual C++ project. The only real challenge here is making it easy to read and edit with minimal fuss.

What's particularly nice about Waveform is that it uses all free libraries, so I've actually been able to get away with using Fedora's development packages with no additional include directories needed. The incompatibility is only limited to Windows-dependent code (use of the Windows API, WGL display, etc.).

2. Fix the basic C/C++ things that g++ doesn't like

This is also a relatively simple task, but it can take up lots of time. In fact, it took up most of the first week of work. The problem is that Microsoft Visual C++ lets you get away with tons of things that break compatibility, aren't in the C/C++ standard, or just flat-out makes no sense. This can range to silly things like #include <gl\gl.h> rather than #include <GL/gl.h> to bizarre things like this:

lower.h...

#include "upper.h"
enum HERP_LEVEL {
	LOW,
	MEDIUM,
	HIGH
};

upper.h...

struct hypothetical {
	HERP_LEVEL msvc = HIGH;
};

Basically, in MSVC++ a file is apparently (in some cases?) able to use things declared/defined in files that include it, rather than the other way around. What.

While fixing this I also flag, in the Makefile, files that contain code that cannot be fixed simply by making it compliant with g++ (the failures that are the most fun are the walls of "DWORD not defined" errors). Until I fix those specifically, they're commented out in the build process.

Once we get to the bottom of the Makefile without explosions, it's time to get into the "real" porting...

3. Port Win32 threading to pthreads

I just started getting into this yesterday, so this is where the super-in-depth details stop, unfortunately. I will say that I am attempting to port the threading code to SDL, as this will prevent from having to maintain two separate blocks of threading code covered in #ifdefs. Considering I will be using SDL for display management and input, this seemed like the most logical solution.

4. Cross-platform display management and input

SDL erry day, plain and simple. These will probably be the simplest tasks, as SDL does both of these things beautifully with as little fuss as possible, at least in my experience.

5. Port AVI playback to a cross-platform solution (TheoraPlay?)

Right now you will notice in your Waveform Steam installation that the game as a whole is about 900MB. Want to discover something a bit odd? Go into steamapps/waveform/movies.

How do you like the size of that file, eh?

Another problem with the engine is that the movie playback is currently Windows-dependent. Well, how might we solve this problem? Well, icculus recently released TheoraPlay, so why not kill two birds with one stone, right? Well, there's currently one problem (and an ironic one at that): TheoraPlay is currently Unix only! So, if we were to use this, it would only be usable on Mac and Linux, which would once again be a pile of #ifdefs and would cause the assets to differ between platforms, which is unacceptable.

I may attempt to port this myself, or just port the AVI playback, but I'm hoping that someone is working on porting TheoraPlay to Windows. If you think you could do this, I'd really appreciate it (as would many others, I'm sure)! Heck, I'll even try to get your name in the credits in some form (I imagine Eden would be just as thrilled to have it as well, so there you go).

6. OSX/Linux crash reporting (stack walker, e-mail reports, etc.)

The Waveform engine has crash reporting, but it's Windows-only at the moment. Not much to say here, really.

7. Debug/Test the crap out of everything after it finally compiles

Perhaps the scariest part about porting something to a new platform is that most (if not all) of this is done before you even get it running on the target operating system. Even worse, you also have to make sure that the changes you made to the source don't negatively affect the original versions, either. Once everything is compiled, it's time to test it on every setup you can possibly build, fix any outstanding bugs (hopefully just bugs you created, which is a scary thing to say) and prepare it for public release. Depending on how drastic these changes are I may try to have closed beta testing, but as I've said before, I'll have to clear it with Eden first.


And there you have it, Internet! That's my current game plan for the next... erm... well, that's something I must also discuss.

There is no announced release date for either platform. I will say that the Mac version will probably be released first (though I may try and do unsupported Linux beta builds if they exist at the time of the Mac release) and we have a deadline that we want to hit, but we still don't know if it's a realistic goal at this time. As it turns out there may be someone else joining the team who is familiar with Mac/Linux development as well, so if something starts kicking my butt I may nag him into helping me out.

And there you have it, for real this time! Thanks for reading this extravagant wall of text; I do hope it was at least mildly interesting. Later posts will probably not be this long, but they'll be about much more specific topics. See you around!

[Cheeseness]

Thanks for the update! It's awesome to hear how you're going.

[flibitijibibo]

Time for another update!

So let's get right into it: Where are we, two weeks later?

1. Create Unix Makefile based on VCProject - Done!
2. Fix the basic C/C++ things that g++ doesn't like - Done!
3. Port Win32 threading to pthreads - Done!
4. Cross-platform display management and input - In progress
// 5. Port AVI playback to a cross-platform solution (TheoraPlay?)
// 6. OSX/Linux crash reporting (stack walker, e-mail reports, etc.)
7. Fix remaining Windows-y problems, Steamworks
8. Debug/Test the crap out of everything after it finally compiles

3. Port Win32 threading to pthreads

The first thing I did after the last update was port the threading to SDL. This one, while not taking *too* long, was admittedly about as awkward as... well, threading.

The thing is, no matter what library/API you use for threading, odds are your implementation of threading is going to contain several h4x in it. It's not really the programmer's fault (at least not the one making the engine...), as threading is just a naturally dangerous area in the world of software design. It can be summed up nicely in this tweet, really.

So now that we've established that threading is dangerous, imagine porting a threading implementation to an entirely different library.

It's things like this that make me very appreciative of emulator developers. Very smart bunch, they are.

In fear of causing the Windows build to break, I spent most of the time trying to emulate the original threading with SDL, only making "changes" when I knew it was going to be better/safer that way (SDL_CondSignal(), for instance, made a rather large function a one-liner with no problems). Of course, emulation isn't exactly easy... In a nutshell (not a transcription):

"Hey, I just pushed a commit attempting to port threading to SDL."
"Oh neat, I'll build it and see wOh god what is this even"
"Yeah, sorry... :C"

Luckily, Ryan was able to get it working again based on my SDL port. He was very proud, and rightly so.

4. Cross-platform display management and input

I actually just finished a first draft of an SDL port of display management, so I'll get back to you on what happens with this. There wasn't a whole lot to it other than a few instances of this:

1. Read function, 100+ lines
2. Double check the SDL manual
3. Sit there for 2+ minutes in doubt
4. Delete whole function, replace with 2 SDL functions
5. Sit there in more doubt wondering if you forgot something huge
6. Screw it, engage danger mode. Next function!

I guess that's why they call it the Simple Directmedia Layer, hm?

So up next will be input. Again, I'll get back to you once I've gotten into that more.

// 5. Port AVI playback to a cross-platform solution (TheoraPlay?)
// 6. OSX/Linux crash reporting (stack walker, e-mail reports, etc.)

As it turns out, these two will not necessarily be needed for the ports to work. The AVI movie playback has actually been disabled in the current version of Waveform and Steamworks handles the crash reporting at the moment. Linux, of course, won't be able to take advantage of that (at least not for now... pretty please? ), but I should be able to work with you guys pretty easily if you have technical issues with the Linux version.

7. Fix remaining Windows-y problems, Steamworks

flibitFile.cpp:1337:3: error: DWORD was not declared in this scope

Other than that, it's just making sure the game works without the need for Steamworks. DLC is the only really big deal here, since it actually pertains to game content.

8. Debug/Test the crap out of everything after it finally compiles

So it probably looks like there's not a lot left, right? Well, there aren't a lot of files left to get compiling, but there still needs to be LOTS of testing. We're hoping that it works out of the box once it's done, but that basically never happens* with ports in my experience, so if we need help we will likely ask for beta testers. If you want to do this, make sure you've posted your specs here so we know you're interested!


That about covers everything so far. Considering what's left on my TODO list there will probably be an update sooner than later, so keep an eye out for that one. Odds are it will contain either beta information or a release date, if all goes well!


*- That is indeed another port I'm working on. Will write about it elsewhere when it's ready.

[Cheeseness]

Awesome update
I'm still surprised by how quickly you're able to pull things together.

[XYTN]

My system:

MacBook Pro
17-inch, Early 2011
2.3 GHz Intel Core i7
8GB RAM
AMD Radeon HD 6750M

[flibitijibibo]

Finally, a post that's not just an update! Have a request for the Linux users out there:

I need installation commands for various distributions. Looking for the following libraries on various distros:
- GLEW
- OpenAL Soft
- FreeALUT
- SDL
- SDL_image
- Vorbisfile

Here's the Fedora command, for instance:

yum install libGLEW openal-soft freealut SDL SDL_image libvorbis

Ubuntu/Mint/Debian (with some help from Kevin Lange and the posts below):

apt-get install libglew1.6 libopenal1 libalut0 libsdl1.2debian libsdl-image1.2 libvorbisfile3

Arch (thanks meklu):

pacman -S extra/glew extra/openal extra/freealut extra/sdl extra/sdl_image extra/libvorbis

Gentoo (thanks meklu):

emerge media-libs/glew media-libs/openal media-libs/freealut media-libs/libsdl media-libs/sdl-image media-libs/libvorbis

Thanks in advance!

[Bestia]

This are my specs:

Kernel: Linux 3.2.0-25-generic x86_64
Distro: Ubuntu 12.04 LTS
CPU:    AMD Athlon(tm) 64 X2 Dual Core Processor 4400+
             2 cores/threads
             2300.00 MHz
RAM:    1.95 GiB
GPU:    GeForce 9600 GT/PCIe/SSE2
              OpenGL 3.3.0 NVIDIA 295.49
              GLSL 3.30 NVIDIA via Cg compiler

Everything you posted in command for Ubuntu is still relevant here on 12.04 LTS. I'm not sure is the OpenAl Soft installed by default. If you want to install it on Ubuntu then it's

sudo apt-get install libopenal1

I also don't know if you are planning 64-bit release or only 32-bit. If it's the latter keep in mind that you will need to install relevant i386 versions of those libs.

[xpander69]

Mint 13 or ubuntu 12.04 have the same.
you are just missing openal

apt-get install libopenal1 libalut0 libglew1.6 libsdl1.2debian libfreeimage3 libvorbisfile3

specs:

Host: xpander-desktop
Kernel: Linux 3.2.0-24-generic x86_64
Distro: Linux Mint 13 Maya
CPU: AMD Phenom(tm) II X4 955 Processor
4 cores/threads
3200.000 MHz
RAM: 7.79 GiB
GPU: GeForce GTX 560 Ti/PCIe/SSE2
OpenGL 4.2.0 NVIDIA 295.53
GLSL 4.20 NVIDIA via Cg compiler

[flibitijibibo]

Awesome, thanks guys.

Edit: Oh, forgot to mention this! I have both 64-bit and 32-bit builds of the game, so yeah. Won't have to worry about that for Waveform.