What it is, what gear to use, and what to listen to for...
Mastering of audio material is a very specific discipline. As a matter of fact it is so specific that there are dedicated mastering engineers out there who don't provide any other types of pro-audio services in their line of work. After having worked in pro-audio field for the past twenty-five years, I can confirm that experienced set of ears along with knowledge of most modern trends in the audio industry play a big role in providing client with high quality results. In addition, one needs mastering-specific tools in order to have the flexibility of tweaking client's mixes accordingly and since most clients these days mix their recordings on their own computers, mastering engineer is expected to act as a wizard of all sorts in order to eliminate many mistakes and irregularities which often come out of this type of workflow (be it balance of individual instruments, or their color, or the density of sound produced on average). The truth of the past remains still today - better the mix, easier the mastering of it.
INTRODUCTION TO MASTERING FUNDAMENTALS
Most mastering engineers of the past twenty years or so have owned a high quality digital audio workstation with outstanding and reliable sound card, pro-quality AD/DA converter, up/down-sampling decoder/encoder, and even digital external outboard gear collection (for EQ, Reverb, Dynamics, Limiting, and more) in addition to the many types of internal plugins which substitute or replace physical equipment and are often bundled or purchased separately via online retailers such as Waves, Sony, McDSP, Digidesign, Avid, and many more.
You may have also heard of loudness wars where each new release seemed to have been louder than the one before because simply most humans perceive loudness with quality - louder it sounds, better it sounds when two tracks are compared with each other. It's a trick played by the nature on our hearing and takes a long time to get used to controlling. Fortunately, since the invention of iTunes and iTunes Radio standard (among other online streaming services), new loudness measuring tools (such as LUFs meter) were introduced and because of them, mastering engineers no longer have to guess how loud their masters should be. This plain leveling field is very important as it gives us all a reference point to be used when finalizing our masters.
For example, just last month (05-2018) I ran a comparison of over dozen 2016-2018 CD releases of various artists and labels from acoustic jazz solo and combo releases, through folk and country recordings, rock and blues genres, as well pop music, such as Adele's 25. While most iTunes recordings have a recommendation in the rage of negative 13-15 LUFs, some rock and pop CDs were -12 to -11 LUFs or louder thus noticeably pushing the loudness threshold beyond reasonable limits. One can also see all transients present in these 16bit/44.1kHz CD rips and how they have been treated during mastering stage (are they dynamic or squashed, consistent or not, limited softly or via hard knee and how do they affect the overall sound and the peaks). All these "scientifically reversed engineering experiments" I've been conducting for years have helped me get a complete picture where the music industry standards stand and how one's mixes and masters compare to the very best productions out there.
ANALOG OR DIGITAL MASTERING?
There is actually a pretty straight forward answer to this question as explained below. Know that this recommendation is based on my personal experiences of working with both analog and digital concepts since the mid 1990s, so even though I feel confident that my recommendations have a merit, it is also true that our knowledge and musical tastes keep expanding and changing as we grow older and more experienced and as new technologies and workflows are introduced by the recording industry players. Yet, it is possible to achieve outstanding results via various means and techniques as long as we let our experiences guide our ears accordingly.
1. When the original material is recorded using strictly analog means, such as analog mixing board, analog outboard effects (tube or solid state EQs, compressors, mixed-in room sound) and/or in addition is captured onto an analog reel to reel tape before converting and storing its digital copy fully inside the digital domain (projects involving reel to reel tape are rarer to see these days due to the cost of out of print tapes and time consuming maintenance/calibration of tape machines), my initial suggestion would be to stay entirely in the digital domain during the mastering stage. This is because the material recorded and mixed already features the warm sounding analog tube or solid state equipment along with the warmth and natural saturation of analog tape, therefore additional D/A and A/D conversion needed to use analog outboard gear for mastering tasks would diminish the final sound quality produced even when the very best converters are used. Therefore, in my experience, this is the time to use the very best digital outboard gear (where no conversion takes place) and/or plugins at the highest bit rate/sampling frequency possible (often delivered by the mixing engineer at 24bit resolution) and once finished with tweaking the overall sound, can be down-sampled and converted to 16bit/44.1kHz CD standard in the final stage.
2. When using primarily digital equipment to capture the sound of a recording in order to mix it at a later time via digital means (using DAW and its plugins or using digital outboard gear when available with A/D converters of some sort), this is where an analog master can make a huge difference and improve upon the original mix noticeably. Having worked with some of the very best digital and analog outboard gear myself built by legendary gurus of pro-audio inventions such engineers of Neve, Teletronix, Tube-Tech, Neumann, Telefunken, GML, Calrec, Studer, Universal Audio, SSL, Manley, Crane Song, and many more, I've learn to appreciate how their individual sound colors the so often sterile-sounding digital tracks. Most solid state gear will use custom input and output hand-wound transformers known to color the sound of any audio signal in a positive way (while each unit does it differently, depending on the type), and most tube-based gear will provide the necessary warmth associated with acoustically produced tracks of the past sixty+ years. Getting one's hands on working with as much analog equipment available as possible (thanks eBay) and using it on projects while mixing and mastering for clients is extremely beneficial to one's learning.
MASTERING TECHNIQUES AND THEIR CHALLENGES
Here are some simply rules I go by when mastering music (be it for clients or my own CD releases). One can always agree or disagree. As mentioned, each person had a different set of skills, gear to use, experiences and ears to take advantage of and there were many ways which can be utilized to achieve outstanding results. Here is mine:
1. Consults with each client ideally before the mixing stage take place about what type of music, instrumentation, recording space, and mixing facility will be used before a final mix is delivered. Even the best mastering engineer can't substitute for a poorly produced mix, which lacks proper balance and dynamics. Soft mixes have a tendency to sound noisy when mastered to the industry levels, very hot mixes have a tendency to sound distorted, which is impossible to fix in post-production. It is ideal, if the client delivers mixes which are peaking around -3 to -4dBFs with some -25 to -21dBU RMS average loudness or -22LUFs. This gives mastering engineer the proper amount of headroom to work with.
2. Consider stem-mixing. While most mixing houses may not be inclined to this request, stem-mixing gives each client much greater control over the final mix result, which can then be easily tweaked or recreated precisely using these individual stems. Stems are stereo files which feature pre-mixed individual or groups of instruments, already spread in the stereo field the same way they went into the final bus, including all effects imprinted in them. They are then blent with each other in the final mix bus during the summing process. Think of them as sub-groups on an analog mixer. This way, one can improve upon the overall balance of all instruments after the mixing took place if some irregularities are discovered after the mix was finalized.
3. Mastering needs to provide a better sounding result then the original mix. If it doesn't, it is not done properly. When mastering, I'll often run the original mix on a separate stereo bus which I can go to at any given moment to A/B it in real time with what I'm working on during the mastering stage. As mentioned, it is crucial that both A/B-ed stereo tracks have about same loudness level, therefore I often lower the outgoing monitor volume of the mastering track in order to match the relative lower volume of the original mix. That way my ears are not fooled easily since louder always sounds better as already explained above. With really good mixes, one can apply EQ/Compression/Limiting to the entire album across the board so to speak however with uneven or unbalanced tracks/mixes, one has to work on each song individually and therefore spend much more time tweaking and making it all sound leveled and balanced - something, which the mixing engineer was supposed to do him/her self at first place.
4. While most mastering can be applied to the entire Left/Right field spectra simultaneously, there are often instruments in the middle of the stereo field which need our help (most often bass, snare, and kick). Therefore, having the ability to use Mid/Side mastering technique is of great help, when one wants to tweak the center channel vs. both side channels. While there are quite a few plugins nowadays who serve this very purpose (Isotope's Ozone comes to mind), my all time favorite still today (after running many A/B tests and having worked with it for the past 15+ years) is a master bundle by TC Electronics in their high end TC6000 effects processor, which I'm a proud owner of. This mastering suite has a very powerful MD4 engine, which provides me not only with M/S EQ section, but also M/S compression section with up to five individual bands of compression to be chosen from. Last portion of this bundle is a very high quality limiter, which offers quite a few options for limiting the peaks, either in less or more aggressive way. Besides it's reverbs are world-famous and to my ears still among the very best sounding out there. I use them on all my mixing sessions.
5. Once the high resolution master is completed, the one before last stage of mastering is down-sampling to a CD format rates of 16bit/44.1kHz. This process too can be divided to either in-the-box experience (many software developers offer plugins which can down-sample high resolution files with a decent to high quality) or via digital outboard gear which often provides higher audio quality experience but it is much more costly and time-consuming to do. My favorite hardware for this function is a combination of Prism Sound ADA-XR 24bit to 16bit reduction algorithms built into the unit itself (four dithering curves to choose from) and frequency converter by Lavry aka DB Technologies known as 3000S.
6. Finally, when final master WAV tracks are finalized at 16/44.1, one needs to use a specialized software to print the final CD master onto a CD-R media before delivering it to a mastering facility. The master CD-R should include CD text and information about the owner, title, and last but very important ISRC codes (International Standard Recording Code) intended to protect the media and its creator. Sony's CD Architect (ideal for PC users) is a good choice still today since it also has the ability to provide quesheet, although there are a few more modern programs available these days as well.