Essbase Performance Series: Part 3 – Local Storage Baseline (Anvil)

Welcome to part three of a series that will have a lot of parts.  In our we took a look at our test results using the CrystalDiskMark synthetic benchmark.  Today we’ll be looking at the test results using a synthetic benchmark tool named Anvil.  As we said in part two, the idea here is to see first how everything measures up in synthetic benchmarks before we get into the actual benchmarking in Essbase.

Also as discussed in part two, we have three basic local storage options:

• Direct-attached physical storage on a physical server running our operating system and Essbase directly
• Direct-attached physical storage on a virtual server using VT-d technology to pass the storage through directly to the guest operating system as though it was physically connected
• Direct-attached physical storage on a virtual server using the storage as a data store for the virtual host

As we continue with today’s baseline, we still have the following direct-attached physical storage on the bench for testing:

• One (1) Samsung 850 EVO SSD (250GB)
  • Attached to an LSI 9210-8i flashed to IT Mode with the P20 firmware
  • Windows Driver P20
• Four (4) Samsung 850 EVO SSD’s (250GB)
  • Attached to an LSI 9265-8i
  • Windows Driver 6.11-06.711.06.00
  • Configured in RAID 0 with a 256kb strip size
• One (1) Intel 750 NVMe SSD (400GB)
  • Attached to a PCIe 3.0 8x Slot
  • Firmware 8EV10174
  • Windows Driver 1.5.0.1002
  • ESXi Driver 1.0e.1.1-1OEM.550.0.0.139187
• Twelve (12) Fujitsu MBA3300RC 15,000 RPM SAS HDD (300GB)
  • Attached to an LSI 9265-8i
  • Windows Driver 6.11-06.711.06.00
  • Configured three ways:
    • RAID 1 with a 256kb strip size
    • RAID 10 with a 256kb strip size
    • RAID 5 with a 256kb strip size

So what benchmarks were used?

• CrystalDiskMark 5.0.2 (see part two)
• Anvil’s Storage Utilities 1.1.0.337

And the good stuff that you skipped to anyway…benchmarks!

Now that we’ve looked at CrystalDiskMark, we’ll take a look at Anvil results.  While Anvil results include reads and writes in megabytes per second, we’ll instead focus on Inputs/Outputs per Second (IOPS).  Here we see that the Intel 750 is keeping pace nicely with the RAID 0 SSD array.  In this particular test, even our traditional drives don’t look terrible.

Next up we’ll look at the random IOPS performance.  So much for our traditional drives.  Here we really see the power of SSD’s versus old-school technology.  It is interesting that all three solutions hover pretty closely together.  But this is likely a queue depth issue.

Let’s see how things look with a queue depth of four.  Things are still pretty clustered here, but much higher across the board.

And now for a queue depth of 16.  Now this looks better.  The Intel 750 has, for the most part, easily outpaced the rest of the options.  The RAID 0 SSD array looks pretty good here as well.

That’s it for the read tests.  Next we move on to the write tests.  We’ll again start with the sequential writes.  Before we expand our queue depths, the RAID 0 SSD array is looking like the clear winner.

Our random write test seems to follow closely to our random read test.  The Intel 750 is well in the lead with the other SSD options trailing behind.  Also of interest, the Intel 750 seems to struggle when physically attached and as a data store in these tests. We’ll see if this continues.

When the queue depth increases, to four, we see the Intel 750 continue to hold its lead.  The RAID 0 SSD array is still trailing the regular single SSD.  As with the previous random test, the Intel 750 continues to struggles, though the physical test has improved.

Finally, we’ll check out the queue depth at 16.  It looks like our physical Intel 750 has finally caught up to the passthrough.  This feels like an odd benchmark result, so we’ll see how this looks in real Essbase performance.  We also finally see that the RAID 0 SSD array has pulled ahead of the single drive by a large margin.

Next up…we’ll start taking a look at the actual Essbase performance for all of these hardware choices. That post is a few weeks away with Kscope rapidly approaching.

Essbase Performance Series: Part 2 – Local Storage Baseline (CDM)

Welcome to part two of a series that will have a lot of parts.  In our introduction post, we covered what we plan to do in this series at a high level.  In this post, we’ll get a look at some synthetic benchmarks for our various local storage options.  The idea here is to see first how everything measures up in benchmarks before we get into the actual benchmarking in Essbase.

As we discussed in our introduction, we have three basic local storage options:

• Direct-attached physical storage on a physical server running our operating system and Essbase directly
• Direct-attached physical storage on a virtual server using VT-d technology to pass the storage through directly to the guest operating system as though it was physically connected
• Direct-attached physical storage on a virtual server using the storage as a data store for the virtual host

For the purposes of today’s baseline, we have the following direct-attached physical storage on the bench for testing:

• One (1) Samsung 850 EVO SSD (250GB)
  • Attached to an LSI 9210-8i flashed to IT Mode with the P20 firmware
  • Windows Driver P20
• Four (4) Samsung 850 EVO SSD’s (250GB)
  • Attached to an LSI 9265-8i
  • Windows Driver 6.11-06.711.06.00
  • Configured in RAID 0 with a 256kb strip size
• One (1) Intel 750 NVMe SSD (400GB)
  • Attached to a PCIe 3.0 8x Slot
  • Firmware 8EV10174
  • Windows Driver 1.5.0.1002
  • ESXi Driver 1.0e.1.1-1OEM.550.0.0.139187
• Twelve (12) Fujitsu MBA3300RC 15,000 RPM SAS HDD (300GB)
  • Attached to an LSI 9265-8i
  • Windows Driver 6.11-06.711.06.00
  • Configured three ways:
    • RAID 1 with a 256kb strip size
    • RAID 10 with a 256kb strip size
    • RAID 5 with a 256kb strip size

So what benchmarks were used?

• CrystalDiskMark 5.0.2
• Anvil’s Storage Utilities 1.1.0.337

And the good stuff that you skipped to anyway…benchmarks!

We’ll start by looking at CrystalDiskMark results.   The first result is a sequential file transfer with a queue depth of 32 and a single thread.  There are two interesting results here.  First, our RAID 10 array in passthrough is very slow for some reason.  Similarly, the Intel 750 is also slow in passthrough.  I’ve not yet been able to determine why this is, but we’ll see how it performs in the real world before we get too concerned.  Obviously the NVMe solution wins overall with our RAID 0 SSD finishing closely behind.

Next we’ll look at a normal sequential file transfer.  We’ll see here that all of our options struggle with a lower queue depth.  Some more than others.  Clearly the traditional hard drives are struggling along with the Intel 750.  The other SSD options however are much closer in performance.  The SSD RAID 0 array is actually the fastest option with these settings.

Next up is a random file transfer with a queue depth of 32 and a single thread.  As you can see, on the random side of things the traditional hard drives, even in RAID, struggle.  Actually, struggling would probably be a huge improvement over what they actually do.  The Intel 750 takes the lead for the physical server, but it actually gets overtaken by the RAID 0 SSD array for both of our virtualized tests.

Our final read option is a normal random transfer. Obviously everything struggles here.  A big part of this is just not having enough queue depth to take advantage of the potential of the storage options.

Next we will take a look at the CrystalDiskMark write tests.  As with the read, we’ll start with a sequential file transfer using a queue depth of 32 and a single thread.  Here we see that the RAID 0 SSD array takes a commanding lead.  The Intel 750 is still plenty fast, and then the single SSD rounds out the top three.  Meanwhile, the traditional disks are still there…spinning.

Let’s look at a normal sequential file transfer.  For writes, our traditional drives clearly prefer lower queue depths.  This can be good or bad for Essbase, so we’ll see how things look when we get to the real-world benchmarks.  In general, our top three mostly hold with the RAID 0 traditional array pulling into third in some instances.

On to the random writes.  We’ll start off the random writes with a queue depth of 32 and a single thread.  As with all random operations, the traditional disks get hammered.  Meanwhile, the Intel 750 has caught back up to the RAID 0 SSD array, but is still back in second place.

And for our final CrystalDiskMark test, we’ll look at the normal random writes.  Here the Intel 750 takes a commanding lead while the RAID 0 SSD array and the single SSD look about the same.  Again, more queue depth helps a lot on these tests.

In the interest of making my blog posts a more reasonable length, that’s it for today.  Part three of the series will be more baseline benchmarks with a different tool to measure IOPS (I/O’s per second), another important statistic for Essbase.  Then hopefully, by part four, you will get to see some real benchmarks…with Essbase!

Part five teaaser:

Essbase Performance Series: Part 1 – Introduction

Welcome to the first in a likely never-ending series about Essbase performance.  To be specific, this series will be designed to help understand how the choices we make in Essbase hardware selection affect Essbase performance.  We will attempt to answer questions like Hyper-Threading or not, SSD’s or SAN, Physical or Virtual.  Some of these things we can control, some of them we can’t.  The other benefit of this series will be the ability to justify changes in your organization and environment.  If you have performance issues and IT want’s to know how to fix it, you will have hard facts to give them.

As I started down the path of preparing this series, I wondered why there was so little information on the internet in the way of Essbase benchmarks.  I knew that part of this was that every application is different and has significantly different performance characteristics.  But as I began to build out my supporting environment, I realized something else.  This is a time consuming and very expensive process.  For instance, comparing Physical to Virtual requires hardware that is dedicated to the purpose of benchmarking.  That isn’t something you find at many, if any clients.

As luck would have it, I have been able to put together a lab that allows me the ability to do all of these things.  I have a dedicated server for the purpose of Essbase benchmarking.  This server will go back and forth between physical and virtual and various combinations of the two.  Before we get into the specifics of the hardware that we’ll be using, let’s talk about what we hope to accomplish from a benchmarking perspective.

There are two main areas that we care about that relate to Essbase performance.  First, we have the back-end performance of Essbase calculations.  When I run an agg or a complex calculation, how long does it take?  Second, we have the front-end performance of Essbase retrieves and calculations.  This is a combination of how long end-user queries take to execute and how long user-executed calculations take to complete.  So what will we be testing?

Storage Impact on Back-End Essbase Calculations

We’ll take a look at impact our options in storage have on Essbase calculation performance.  Storage is our slowest bottleneck, so we’ll start here to find the fastest solution that we can use for the next set of benchmarks.  We’ll compare each of our available storage types three ways: a physical Essbase server, a virtual Essbase server using VT-d and direct attached storage, and a virtual Essbase server using data stores.  Here are the storage options we’ll have to work with:

• Samsung 850 EVO SSD (250GB) on an LSI 9210-8i
• Four (4) Samsung 850 EVO SSD’s in RAID 0 on an LSI 9265-8i (250GB x 4)
• Intel 750 NVMe SSD (400GB)
• Twelve (12) Fujitsu MBA3300RC 15,000 RPM SAS HDD (300GB x 12) in RAID 1, 1+0, and RAID 5

CPU Impact on Back-End Essbase Calculations

Once we have determined our fastest storage option, we can turn our attention our processors.  The main thing that we can change as application owners is the Hyper-Threading settings.  Modern Intel processors found in virtually all Essbase clients support it, but conventional wisdom tells us that this doesn’t work out very well for Essbase.  I would like to know what the cost of this setting is and how we can best work around it.  ESXi (by far the most common hypervisor) even gives us some flexibility with this settings.

Storage Impact on Front-End Essbase Query Performance

This one is a little more difficult.  Back-End calculations are easy to benchmark.  You make a change, you run the calculation, you check the time it took to execute.  Easy.  Front-End performance requires user interaction, and consistent user interaction at that.  So how will we do this?  I can neither afford load runner, nor have the time to attempt to learn this complex tool.  Again, as luck would have it, we have another option.  Our good friends at Accelatis have graciously offered to allow us to use their software to perform consistent front-end benchmarks.

Accelatis has an impressive suite of performance testing products that will allow us to test specific user counts and get query response times so that we can really understand the impact of end-user performance.  I’m very excited to be working with Accelatis.

CPU Impact on Front-End Essbase Query Performance

This is an area where we can start to see more about our processors.  Beyond just Hyper-Threading, which we will still test, we can look at how Essbase is threading across processors and what impact we can have on that.  Again, Accelatis will be key here as we start to understand how we really scale Essbase.

So what does the physical server look like that we are using to do all of this?  Here are the specs:

You can see specs of the full lab supporting all of the testing here.  And now, because I promised benchmarks, here are a few to start with:

Physical Server, Samsung EVO 850 x 4 in RAID 0 on an LSI 9265-8i

Physical Server, Intel 750 NVMe SSD

Well…that’s fast.  In our next post in the series, we’ll look at benchmarking all of our baseline storage performance for Physical, Virtual with VT-d, and Virtual with Data stores.  This will be our baseline for the next post after that about actual Essbase performance.  In the meantime, I’ll also be working towards getting access to some fast network storage to test that against all of our direct and virtual options.  In the meantime, let’s try out a graph and see how it looks:

CrystalDiskMark 5.0.2 Read Comparison:

CrystalDiskMark 5.0.2 Write Comparison: