My First FreeNAS: Part 2 – Install, Test, and Configure

Re-Introduction

It has been quite a while since my last post on my new FreeNAS build.  This project was placed on the back-burner while I had a lot going on.  I’m finally starting to get everything stabilized, so now I’m back at working on my new FreeNAS box.  You can view part 1 of this series here, but just as a quick re-cap, let’s talk about the system specs.  Here is a revised list (things in bold have been changed from Part 1):

• SuperChassis 846TQ-R900B (with Supermicro 1200W model PWS-1K21P-1R)
• (2) E5-2670 @ 2.6 GHz
• Supermicro X9DR7-LN4F-JBOD
• 256GB Registered ECC DDR3 RAM (16 x 16GB)
• Noctua i4 Heatsinks
• (5) Noctua NF-R8 (to bring the noise level down on the chassis)
• (2) SanDisk Cruzer 16GB CZ33
• (2) Intel S3500 80GB SSD’s
• (1) Supermicro AOC-2308-l8e
• (3) Full-Height LSI Backplates (for the AOC-2308’s and the P3605)
• (6) Mini-SAS Breakout Cables
• Intel P3605 1.6TB PCIe SSD
• (9) 2TB HGST Ultrastar 7K3000 Hard Drives
• (4) 10Gb Twinax DAC Cables
• (2) Intel X520-DA2

So why did I make these changes?  The power supply was a no-brainer.  The R900B chassis was really, really loud.  I have another Supermicro 846 with a PWS-1K21P which is totally tolerable.  So I went ahead and picked one of those up from Ebay for pretty cheap.  I decided to get rid of the USB drives and replaced them with S3500’s.  I’ve had two USB sticks go bad on me in the last two months and was having some issues during the installation, so I found a pair of new S3500’s a great price.

Finally, I decided to use the eight (8) SATA 3G ports on the motherboard so that I could save a PCIe slot for the controller.  I just hooked up classic hard drives to those ports, which can’t even come close to saturating the 3G speeds.  This lets me add another PCIe SSD or an external SAS card to add more drives in an expander.

FreeNAS Installation

Now that the system is finally built and has gone through the burn-in process, it’s time to install FreeNAS.  This is a pretty simple process that is very well documented, so I won’t bore you with my screenshots.  Instead, go check out the FreeNAS site here:

FreeNAS Installation Documentation

I did install directly to a mirrored set of drives, which as you will read in the documentation is as easy as pressing the spacebar!

Once you get through the installation process, it will tell you the IP address you should use to go access the web-based GUI.  This is where the fun really starts.  Again, there are guides for everything that I did, so I’ll instead run through my high-level steps and provide links to the resources that I made use of.

Final Burn-In

Before I made it to the actual configuration of FreeNAS, I had to finish my burn-in process.  In part 1, we completed the burn-in on the majority of our hardware, but now we have to burn in the hard drives.  If you are using all SSD’s, you don’t have to worry about this.  If you have hard drives, even brand new hard drives, you should burn them in.  Luckily, there’s a great burn-in guide on the FreeNAS forums:

FreeNAS Burn-In Guide

FreeNAS Initial Configuration

Now that everything is tested and burn-in has been completed was set up a storage volume.  You really need at least one of these configured before you can move on to the rest of the steps.  I highly recommend reading up on ZFS if you haven’t already.  There’s a great guide on the FreeNAS forums for those of you completely new to ZFS.  Check it out here:

ZFS Primer Presentation

I set up a pair of volumes to start.  First I set up my classic hard drive volume.  I chose to use a striped set of mirrored disks to give me the best performance and redundancy for my benchmarking use.  This is basically what hardware RAID would could RAID 10.  I will use this volume to test network storage for my Essbase benchmarks.  I’ve also set up a single volume using my 1.6TB PCIe SSD.  Just to break up the text…you may remember this from part 1:

Next I moved on to getting my network configuration up and running.  For me, this was setting up the global settings like DNS servers and the Gateway.  Then I moved on to set up the networking for my server.  Basically I added my main network interface and changed the IP to a static address.  Both of these topics are covered here:

FreeNAS Network Configuration Documentation

Now that the basic network configuration is complete, we can move on to a slightly  This one isn’t covered as well in the FreeNAS documentation, but a community member has written an excellent guide that I found very helpful.  As a bonus, it also helps you set up your first CIFS share for Windows users on your domain to share:

FreeNAS Active Directory Configuration Documentation

FreeNAS Advanced Configuration

And with that, we can finally move on to having FreeNAS support the on-going benchmarking effort!  There are two main network storage technologies that are by far the most common in the Hyperion EPM world:  NFS and iSCSI.  I started with NFS and created a pair of datasets: one on my hard drive volume and one on my NVMe volume.  I used this guide:

FreeNAS NFS Configuration with ESXi

The guide is a little out of date, but the main thing to remember is that you need to set the maproot property, or ESXi will not work properly.  I also have my root passwords set the same, which is a requirement.  I’m sure there is a better, more secure way to do this, but on my home lab, I’m content.

Once I completed the configuration of my NFS shares, I turned my attention to iSCSI.  NFS requires basically a single screen to set everything up.  iSCSI on the other hand, is a bit more involved.  I followed this guide:

FreeNAS iSCSI Configuration with ESXi

I also found another guide that has more detail, albeit older, and is from a much more trusted source:

Older FreeNAS iSCSI Configuration with ESXi

Conclusion and Next Steps

Now that I’ve made it through my initial setup of FreeNAS, I’ve had an opportunity to run a few quick benchmarks.  Here’s a benchmark of one of my NFS shares:

And here’s one of my iSCSI shares:

I’ll have a lot more information on this in my next post.  We’ll dive into benchmarking each of the configurations and make a variety of changes to improve performance.  It should make for an interesting read for the truly nerdy out there!

My First FreeNAS: Part 1 – Build and Burn-In

Kscope16 is over, my parts have arrived, and its finally time to start my FreeNAS build.  Today I’m going to run through my actual build process and the start of my burn-in process.  Let’s start with the hardware…what did I order again?

• SuperChassis 846TQ-R900B
• (2) E5-2670 @ 2.6 GHz
• Supermicro X9DR7-LN4F-JBOD
• 256GB Registered ECC DDR3 RAM (16 x 16GB)
• Noctua i4 Heatsinks
• (5) Noctua NF-R8 (to bring the noise level down on the chassis)
• (2) SanDisk Cruzer 16GB CZ33
• (2) Supermicro AOC-2308-l8e
• (3) Full-Height LSI Backplates (for the AOC-2308’s and the P3605)
• (6) Mini-SAS Breakout Cables
• Intel P3605 1.6TB PCIe SSD
• (9) 2TB HGST Ultrastar 7K3000 Hard Drives
• (4) 10Gb Twinax DAC Cables
• (2) Intel X520-DA2

An here’s the pile of goodies:

I always start with the motherboard by itself:

Next up…the CPU(s):

CPU close-up:

Before we install the heatsinks, let’s install the memory.  The heatsinks are petty big and have a habit of getting in the way:

That’s a lot of memory…how about a close-up:

Now we can install the heatsinks:

Like I said…huge:

Now that we have all of the core components in place on the motherboard, let’s put it into our case:

Obviously, we have a quite a few other components to add (hard drives, add-in cards, etc).  But for now, I like to keep it simple for the burn-in process.  So how do we go about that?  For the basic hardware, there are two recommended steps.  Because memory is so important to FreeNAS, we have to make sure that our memory is in good working order.  For those of us purchasing used hardware, this is especially critical.  Once we have the memory tested, we will then test out our CPU’s to make sure that they are functional and to take a look at the temperatures.

So how do we do this?  You can download utilities like memtest86+ or cpustress and boot up directly using those tools.  But, being that I’m averse to additional work that has already been done by someone else, I just downloaded the latest Ultimate Boot CD.  This comes with a mega-ton of tools including the two I need to start with:  memtest86+ and cpustress.

You can download the ISO here.  Once you have downloaded the ISO, you have two choices.  You can use one of my favorite tools, Rufus, to burn the ISO to a USB thumb drive.  Then you can just boot from the thumb drive.  Your second option is the preferred option.  Hopefully you purchased server-class hardware for your FreeNAS box and that hardware has IPMI and Remote KVM.  If so, then you will likely be able to mount the ISO over the network and easily boot from the virtual media.  This is the option I went for.

My Supermicro board even has two options for this option (options on top of options!).  You can do this through the IPMI interface and mount an ISO from a share or you can use the iKVM to mount the ISO.  Connect to your server with iKVM and select Virtual Media and then Virtual Storage.

Switch to the CDROM&ISO tab, select ISO File from the drop-down, and click Open Image:

Select the ultimate boot CD image name and click Open:

Finally click Plug In:

Once we reboot (or boot up, if you have no other OS installed, it should just boot right in):

We’ll go down to Memory and select Memtest86+:

Memtest86+ is a somewhat newer release of a really old memory testing utility I have used for over a decade: Memtest86.  This release takes the older code and brings support for newer hardware and fixes a number of bugs.  Even still, it is pretty old.  It also takes a long…long time to run with 256GB of memory.  So I ran a single pass to start:

Once that first pass completed (roughly 24 hours), I focused in on stressing my CPU’s.  For this I used cpustress, also included in the Ultimate Boot CD.  I’m less familiar with this stressing tool, as I’ve always been more Windows focused and used tools like Prime95 for this purpose.  Again we boot into the Ultimate Boot CD:

This time we’ll select CPU and then CPUstress:

CPUstress should start up automatically:

This gives us one more menu…I just went with option 1:

Overall, it seems to work pretty well:

Now with that running lets take a look at the CPU temps:

The temps look pretty good for running wide open.  There appears to be headroom for the additional heat that will be generated by the hard drives that will be added to the system.  So how does power usage look?

The numbers look pretty good here.  Again…no drives, so this number will go up considerably by the time we are completely done.  I burned the CPU’s in for a little over 24 hours and then went back to Memtest86+.  I ran that for roughly four more days with no errors.  That’s all for today.  In our next post we’ll finally load up FreeNAS, get our controllers ready to go, and burn in our hard drives.