PBCS Backups with PowerShell: Part 1

I’ve spent a lot of time lately on homelab topics, so I thought I would take a break and put together a post on an EPM topic!  Today we’ll be talking about PBCS backups.

Why Do We Need PBCS Backups

You might be thinking, “Why do I need PBCS backups when Oracle does that for me?”  That’s an excellent question.  The problem is that while Oracle does perform nightly backups of PBCS, they overwrite that backup each night.  So at any given time I only have one backup.  To make things even worse, Oracle has a size limit on your PBCS instance.  That limit is 150GB.  This means that even if we had multiple backups on our pod, we’ll eventually start losing them to the data retention policies.

So what do we do?  We generate a new backup every night and download it to a local server.  The good news is that you almost certainly have a local server already running EPM Automate.  EPM Automate is the automation tool for Oracle’s EPM Cloud suite.  You can use EPM Automate to load data, execute calculations, update meta-data, and…perform backups.  So, we’ve established that we likely need more than a single night of backups, but how many do we need?  This will depend on a few things like the size of your applications and the frequency of change.  For our example, we will keep 30 days of daily backups.

Batch vs. PowerShell

Now that we have determined what we are backing up and how many backups we need to keep, we need to move on to actually performing the backups.  With EPM Automate, we have two commonly used options.  First, we have the old-school method of a batch file.  Batch files are great because they just work and you can find a ton of information on the web about how to do things.  Batch are, however, very limited in their ability to do things like e-mail notifications and remote calls without external tools.  That brings us to PowerShell.  PowerShell is essentially a batch that has the full set of .NET programming capability along with other goodies not directly from .NET.  What does that mean exactly?  That means there is very little I can’t do in PowerShell.

Directory Configuration

Before we configure anything, we need to get a folder structure put together to support scripting, logging, and the actual backup files.  You may already have a structure for your automation processes, but for our example, it will look something like this:

  • C:\Oracle
    • C:\Oracle\Automation
      • C:\Oracle\Automation\Backup
      • C:\Oracle\Automation\Log

EPM Automate Configuration

EPM Automate is a great tool, but we do need to perform a little bit of setup to get going.  For instance, while EPM Automate supports plain text passwords, that wouldn’t pass muster with most IT security groups.  So before we get into PowerShell, let’s encrypt our password.  This is a fairly easy process.  We’ll start up a command prompt and change directory to our EPM Automate bin directory:

cd\
cd Oracle\EPM_Automate\bin

Once we are in the right directory, we can encrypt our password:

epmautomate.bat encrypt YourPasswordGoesHere PickYourKey c:\Oracle\Automation\password.epw

Here are the parameters:

  • Command – the command EPM Automate will execute
    • encrypt
  • Password – the password of the account you plan to use
    • YourPasswordGoesHere
  • Key – you specify anything you want to use to encrypt the password
    • PickYourKey
  • Password File – The full path and file name of the password file that will be generated
    • c:\Oracle\Automation\password.epw

Once we execute the command, we should have our password file so that we can continue.  It should look something like this:

 

Backing Up PBCS with PowerShell

For our first part of this mini-series, we’ll stick with just a basic backup that deletes older backups.  In our next part of the series, we’ll go deeper into error handling and notifications.  Here’s the code..

Path Variables

#Path Variables
$EpmAutomatePath = "C:\Oracle\EPM_Automate\bin\epmautomate.bat"
$AutomationPath = "C:\Oracle\Automation"
$LogPath = "C:\Oracle\Automation\Log"
$BackupPath = "C:\Oracle\Automation\Backup"

We’ll start by defining our path variables.  This will include paths to EPM Automate, our main automation directory, our log path, and our backup path.

Date Variables

#Date Variables
$DaysToKeep = "-30"
$CurrentDate = Get-Date
$DatetoDelete = $CurrentDate.AddDays($DaysToKeep)
$TimeStamp = Get-Date -format "yyyyMMddHHmm"
$LogFileName = "Backup" + $TimeStamp + ".log"

Next we’ll define all of our data related variables.  This includes our days to keep (which is negative on purpose as we are going back in time), our current date, the math that gets us back to our deletion period, a timestamp that will be used for various things, and finally our log file name based on that timestamp.

PBCS Variables

#PBCS Variables
$PBCSdomain = "yourdomain"
$PBCSurl = "https://usaadmin-test-yourdomain.pbcs.us2.oraclecloud.com"
$PBCSuser = "yourusername"
$PBCSpass = "c:\Oracle\Automation\password.epw"

Now we need to set our PBCS variables.  This will include our domain, the URL to our instance of PBCS, the username we’ll use to log in, and the path to the password file that we just finished generating.

Snapshot Variables

#Snapshot Variables
$PBCSExportName = "Artifact Snapshot"
$PBCSExportDownloadName = $PBCSExportName + ".zip"
$PBCSExportRename = $PBCSExportName + $TimeStamp + ".zip"

We’re nearing the end of variables as we define our snapshot specific variables.  These variables will tell us the name of our export, the name of the file that we are downloading based on that name, and the new name of our snapshot that will include our timestamp.

Start Logging

#Start Logging
Start-Transcript -path $LogPath\$LogFileName

I like to log everything so that if something does go wrong, we have a chance to figure it out after the fact.  This uses the combination of our log path and log file name variables.

Log Into PBCS

#Log into PBCS
Write-Host ([System.String]::Format("Login to source: {0}", [System.DateTime]::Now))
&$EpmAutomatePath "login" $PBCSuser $PBCSpass $PBCSurl $PBCSdomain

We can finally log into PBCS!  We’ll start by displaying our action and the current system time.  This way we can see how long things take when we look at the log file.  We’ll then issue the login command using all of our variables.

Create the Snapshot

#Create PBCS snapshot
Write-Host ([System.String]::Format("Export snapshot from source: {0}", [System.DateTime]::Now))
&$EpmAutomatePath exportsnapshot $PBCSExportName

Again we’ll display our action and current system time.  We then kick off the snapshot process.  We do this because we want to ensure that we have the most recent snapshot for our archiving purposes.

Download the Snapshot

#Download PBCS snapshot
Write-Host ([System.String]::Format("Download snapshot from source: {0}", [System.DateTime]::Now))
&$EpmAutomatePath downloadfile $PBCSExportName

Once the snapshot has been created, we’ll move on to downloading the snapshot after we display our action and current system time.

Archive the Snapshot

#Rename the file using the timestamp and move the file to the backup path
Write-Host ([System.String]::Format("Rename downloaded file: {0}", [System.DateTime]::Now))
Move-Item $AutomationPath\$PBCSExportDownloadName $BackupPath\$PBCSExportRename

Once the file has been downloaded, we can then archive the snapshot to our backup folder as we rename the file.

Delete Old Snapshots

#Delete snapshots older than $DaysToKeep
Write-Host ([System.String]::Format("Delete old snapshots: {0}", [System.DateTime]::Now))
Get-ChildItem $BackupPath -Recurse | Where-Object { $_.LastWriteTime -lt $DatetoDelete } | Remove-Item

Now that we have everything archived, we just need to delete anything older than our DateToDelete variable.

Log Out of PBCS

#Log out of PBCS
Write-Host ([System.String]::Format("Logout of source: {0}", [System.DateTime]::Now))
&$EpmAutomatePath "logout"

We’re almost done and we can now log out of PBCS.

Stop Logging

#Stop Logging
Stop-Transcript

Now that we have completed our process, we’ll stop logging

The Whole Shebang

#Path Variables
$EpmAutomatePath = "C:\Oracle\EPM_Automate\bin\epmautomate.bat"
$AutomationPath = "C:\Oracle\Automation"
$LogPath = "C:\Oracle\Automation\Log"
$BackupPath = "C:\Oracle\Automation\Backup"

#Date Variables
$DaysToKeep = "-30"
$CurrentDate = Get-Date
$DatetoDelete = $CurrentDate.AddDays($DaysToKeep)
$TimeStamp = Get-Date -format "yyyyMMddHHmm"
$LogFileName = "Backup" + $TimeStamp + ".log"

#PBCS Variables
$PBCSdomain = "yourdomain"
$PBCSurl = "https://usaadmin-test-yourdomain.pbcs.us2.oraclecloud.com"
$PBCSuser = "yourusername"
$PBCSpass = "c:\Oracle\Automation\password.epw"

#Snapshot Variables
$PBCSExportName = "Artifact Snapshot"
$PBCSExportDownloadName = $PBCSExportName + ".zip"
$PBCSExportRename = $PBCSExportName + $TimeStamp + ".zip"

#Start Logging
Start-Transcript -path $LogPath\$LogFileName

#Log into PBCS
Write-Host ([System.String]::Format("Login to source: {0}", [System.DateTime]::Now))
&$EpmAutomatePath "login" $PBCSuser $PBCSpass $PBCSurl $PBCSdomain

#Create PBCS snapshot
Write-Host ([System.String]::Format("Export snapshot from source: {0}", [System.DateTime]::Now))
&$EpmAutomatePath exportsnapshot $PBCSExportName

#Download PBCS snapshot
Write-Host ([System.String]::Format("Download snapshot from source: {0}", [System.DateTime]::Now))
&$EpmAutomatePath downloadfile $PBCSExportName

#Rename the file using the timestamp and move the file to the backup path
Write-Host ([System.String]::Format("Rename downloaded file: {0}", [System.DateTime]::Now))
Move-Item $AutomationPath\$PBCSExportDownloadName $BackupPath\$PBCSExportRename

#Delete snapshots older than $DaysToKeep
Write-Host ([System.String]::Format("Delete old snapshots: {0}", [System.DateTime]::Now))
Get-ChildItem $BackupPath -Recurse | Where-Object { $_.LastWriteTime -lt $DatetoDelete } | Remove-Item

#Log out of PBCS
Write-Host ([System.String]::Format("Logout of source: {0}", [System.DateTime]::Now))
&$EpmAutomatePath "logout"

#Stop Logging
Stop-Transcript

The Results

Once you execute the PowerShell script, you should see something like this:

Conclusion

There we have it…a full process for backing up your PBCS instance.  The last step would be to set up a scheduled task to execute once a day avoiding your maintenance window.


Build a Homelab Dashboard: Part 8, FreeNAS

My posts seem to be getting a little further apart each week…  This week, we’ll continue our dashboard series by adding in some pretty graphs for FreeNAS.  Before we dive in, as always, we’ll look at the series so far:

  1. An Introduction
  2. Organizr
  3. Organizr Continued
  4. InfluxDB
  5. Telegraf Introduction
  6. Grafana Introduction
  7. pfSense
  8. FreeNAS

FreeNAS and InfluxDB

FreeNAS, as many of you know, is a very popular storage operating system.  It provides ZFS and a lot more.  It’s one of the most popular storage operating systems in the homelab community.  If you were so inclined, you could install Telegraf on FreeNAS.  There is a version available for FreeBSD and I’ve found a variety of sample configuration files and steps.  But…I could never really get them working properly.  Luckily, we don’t actually need to install anything in FreeNAS to get things working.  Why?  Because FreeNAS already has something built in:  CollectD.  CollectedD will send metrics directly to Graphite for analysis.  But wait…we haven’t touched Graphite at all in this series, have we?  No…but thanks to InfluxDB’s protocol support for Graphite.

Graphite and InfluxDB

To enable support for Graphite, we have to modify the InfluxDB configuration file.  But, before we get to that, we need to go ahead and create our new InfluxDB and provision a user.  If you take a look back at part 4 of this series, we cover this in more depth, so we’ll be quick about it now.  We’ll start by logging into InfluxDB via SSH:

influx -username influxadmin -password YourPassword

Now we will create the new database for our Graphite statistics and grant access to that database for our influx user:

CREATE DATABASE "GraphiteStats"
GRANT ALL ON "GraphiteStats" TO "influxuser"

And now we can modify our InfluxDB configuration:

sudo nano /etc/influxdb/influxdb.conf

Our modifications should look like this:

And here’s the code for those who like to copy and paste:

[[graphite]]
  # Determines whether the graphite endpoint is enabled.
  enabled = true
  database = "GraphiteStats"
  # retention-policy = ""
  bind-address = ":2003"
  protocol = "tcp"
  # consistency-level = "one"

Next we need to restart InfluxDB:

sudo systemctl restart influxdb

InfluxDB should be ready to receive data now.

Enabling FreeNAS Remote Monitoring

Log into your FreeNAS via the web and click on the Advanced tab:

Now we simply check the box that reports CPU utilization as a percent and enter either the FQDN or IP address of our InfluxDB server and click Save:

Once the save has completed, FreeNAS should start logging to your InfluxDB database.  Now we can start visualizing things with Grafana!

FreeNAS and Grafana

Adding the Data Source

Before we can start to look at all of our statistics, we need to set up our new data source in Grafana.  In Grafana, hover over the settings icon on the left menu and click on data sources:

Next click the Add Data Source button and enter the name, database type, URL, database name, username, and password and click Save & Test:

Assuming everything went well, you should see this:

Finally…we can start putting together some graphs.

CPU Usage

We’ll start with something basic, like CPU usage.  Because we checked the percentage box while configuring FreeNAS, this should be pretty straight forward.  We’ll create a new dashboard and graph and start off by selecting our new data source and then clicking Select Measurement:

The good news is that we are starting with our aggregate CPU usage.  The bad news is that this list is HUGE.  So huge in fact that it doesn’t even fit in the box.  This means as we look for things beyond our initial CPU piece, we have to search to find them.  Fun…  But let’s get start by adding all five of our CPU average metrics to our graph:

We also need to adjust our Axis settings to match up with our data:

Now we just need to set up our legend.  This is optional, but I really like the table look:

Finally, we’ll make sure that we have a nice name for our graph:

This should leave us with a nice looking CPU graph like this:

Memory Usage

Next up, we have memory usage.  This time we have to search for our metric, because as I mentioned, the list is too long to fit:

We’ll add all of the memory metrics until it looks something like this:

As with our CPU Usage, we’ll adjust our Axis settings.  This time we need to change the Unit to bytes from the IEC menu and enter a range.  Our range will not be a simple 0 to 100 this time.  This time we set the range from 0 to the amount of ram in your system in bytes.  So…if you have 256GB of RAM, its 256*1024*1024*1024 (274877906944):

And our legend:

Finally a name:

And here’s what we get at the end:

Network Utilization

Now that we have covered CPU and Memory, we can move on to network!  Network is slightly more complex, so we get to use the math function!  Let’s start with our new graph and search for out network interface.  In my case this is ix1, my main 10Gb interface:

Once we add that, we’ll notice that the numbers aren’t quite right.  This is because FreeNAS is reporting the number is octets.  Now, technically an octet should be 8 bits, which is normally a byte.  But, in this instance, it is reporting it as a single bit of the octet.  So, we need to multiply the number by 8 to arrive at an accurate number.  We use the math function with *8 as our value.  We can also add our rx value while we are at it:

Now are math should look good and the numbers should match the FreeNAS networking reports.  We need to change our Axis settings to bytes per second:

And we need our table (again optional if you aren’t interested):

And finally a nice name for our network graph:

Disk Usage

Disk usage is a bit tricky in FreeNAS.  Why?  A few reasons actually.  One issue is the way that FreeNAS reports usage.  For instance, if I have a volume that has a data set, and that data set has multiple shares, free disk space is reported the same for each share.  Or, even worse, if I have a volume with multiple data sets and volumes, the free space may be reporting correctly for some, but not for others.  Here’s my storage configuration for one of my volumes:

Let’s start by looking at each of these in Grafana so that we can see what the numbers tell us.  For ISO, we see the following options:

So far, this looks great, my ISO dataset has free, reserved, and used metrics.  Let’s look at the numbers and compare them to the above.  We’ll start by looking at df_complex-free using the bytes (from the IEC menu) for our units:

Perfect!  This matches our available number from FreeNAS.  Now let’s check out df_complex-used:

Again perfect!  This matches our used numbers exactly.  So far, we are in good shape.  This is true for ISO, TestCIFSShare, and TestNFS which are all datasets.  The problem is that TestiSCSI and WindowsiSCSI don’t show up at all.  These are all zVols.  So apparently, zVols are not reported by FreeNAS for remote monitoring from what I can tell.  I’m hoping I’m just doing something wrong, but I’ve looked everywhere and I can’t find any stats for a zVol.

Let’s assume for a moment that we just wanted to see the aggregate of all of our datasets on a given volume.  Well..that doesn’t work either.  Why?  Two reasons.  First, in Grafana (and InfluxDB), I can’t add metrics together.  That’s a bit of a pain, but surely there’s an aggregate value.  So I looked at the value for df_complex-used for my z8x2TB dataset, and I get this:

Clearly 26.4 MB does not equal 470.6GB.  So now what?  Great question…if anyone has any ideas, let me know, as I’d happily update this post with better information and give credit to anyone that can provide it!  In the meantime, we’ll use a different share that only has a single dataset, so that we can avoid this annoying math and reporting issues.  My Veeam backup share is a volume with a single dataset.  Let’s start by creating a singlestat and pulling in this metric:

This should give us the amount of free storage available in bytes.  This is likely a giant number.  Copy and paste that number somewhere (I chose Excel).  My number is 4651271147041.  Now we can switch to our used number:

For me, this is an even bigger number: 11818579150671, which I will also copy and paste into Excel.  Now I will do simple match to add the two together which gives a total of 16469850297712.  So why did we go through that exercise in basic math?  Because Grafana and InfluxDB won’t do it for us…that’s why.  Now we can turn our singlestat into a gauge.  We’ll start with our used storage number from above.  Now we need to change our options:

We start by checking the Show Gauge button and leave the min set to 0 and change our max to the value we calculated as our total space, which in my case is 16469850297712.  We can also set thresholds.  I set my thresholds to 80% and 90%.  To do this, I took my 16469850297712 and multiplied by .8 and .9.  I put these two numbers together, separated by a comma and put it in for thresholds: 13175880238169.60,14822865267940.80.  Finally I change the unit to bytes from the IEC menu.  The final result should look like that:

Now we can see how close we are to our max along with thresholds on a nice gauge.

CPU Temperature

Now that we have the basics covered (CPU, RAM, Network, and Storage), we can move on to CPU temperatures.  While we will cover temps later in an IPMI post, not everyone running FreeNAS will have the luxury of IPMI.  So..we’ll take what FreeNAS gives us.  If we search our metrics for temp, we’ll find that every thread of every core has its own metric.  Now, I really don’t have a desire to see every single core, so I chose to pick the first and last core (0 and 31 for me):

The numbers will come back pretty high, as they are in kelvin and multiplied by 10.  So, we’ll use our handy math function again (/10-273.15) and we should get something like this:

Next we’ll adjust our Axis to use Celsius for our unit and adjust the min and max to go from 35 to 60:

And because I like my table:

At the end, we should get something like this:

Conclusion

In the end, my dashboard looks like this:

This post took quite a bit more time than any of my previous posts in the series.  I had built my FreeNAS dashboard previously, so I wasn’t expecting it to be a long, drawn out post.  But, I felt as I was going through that more explanation was warranted and as such I ended up with a pretty long post.  I welcome any feedback for making this post better, as I’m sure I’m not doing the best way…just my way.  Until next time…


Build a Homelab Dashboard: Part 7, pfSense

After a small break, I’m ready to continue the homelab dashboard series! This week we’ll be looking at pfSense statistics and how we add those to our homelab dashboard. Before we dive in, as always, we’ll look at the series so far:

  1. An Introduction
  2. Organizr
  3. Organizr Continued
  4. InfluxDB
  5. Telegraf Introduction
  6. Grafana Introduction
  7. pfSense

pfSense and Telegraf

If you are reading this blog post, I’m going to assume you have at least a basic knowledge of pfSense. In short, pfSense is a firewall/router used by many of us in our homelabs. It is based on FreeBSD (Unix) and has many available built-in packages. One of those packages just happens to be Telegraf. Sadly, it also happens to be a really old verison of Telegraf, but more on that later. Having this built in makes it very easy to configure and get up and running. Once we have Telegraf running, we’ll dive into how we visualize the statistics in Grafana, which isn’t quite as straight forward.

Installing Telegraf on pfSense

Installation of Telegraf is pretty easy. As I mentioned earlier, this is one of the many packages that we can easily install in pfSense. We’ll start by opening the pfSense management interface:

For most of us, we’re looking at our primary means to access the internet, and as such I would recommend verifying that you are on the latest version before proceeding. Once you have completed that task, you can move on to clicking on System and then Package Manager:

Here we can see all of our installed packages. Next we’ll click on Available Packages:

If we scroll way down the alphabetical list, we’ll find Telegraf and click the associated install button:

Finally, click the Confirm button and watch the installer go:

We should see a success message:

Now we are ready to configure Telegraf. Click on the Services tab and then click on Telegraf:

Ensure that Telegraf is enabled and add in your server name, database name, username, and password. Once you click save, it should start sending statistics over to InfluxDB:

pfSense and Grafana

Now that we have Telegraf sending over our statistics, we should be ready to make it pretty with Grafana! But, before we head into Grafana, let’s make sure we understand which interface is which. At the very least, pfSense should have a WAN and a LAN interface. To see which interface is which (if you don’t know offhand), you can click on Interfaces and then Assignments:

Once we get to our assignments screen, we can make note of our WAN interface (which is what I care about monitoring). In my case, its em0:

Now that we have our interface name, we can head over to Grafana and put together a new dashboard. I’ve started with a graph and selected my TelegrafStats datasource, my table of net, and filtered by my host of pfSense.Hyperion.local and my interface of em0. Then I selected Bytes_Recv for my field:

If you’re like me, you might think that you are done with your query. But, if you take a look at the graph, you will notice that you are in fact…not done. We have to use some more advanced features of our query language to figure out what this should really look like. We’ll start with the derivative function. So why do we need this? If we look at the graph, we’ll see that it just continues to grow and grow. So instead of seeing the number, we need to see the change in the number over time. This will give us our actual rate, which is what the derivative function does. It looks at the current value and provides the difference between that value and the value prior. Once we add that, we should start to see a more reasonable graph:

Our final query should look like this:

Next we can go to our axes settings and set it to bytes/sec:

Finally, I like to set up my table-based legend:

Now let’s layer in bytes_sent by duplicating our first query:

And our final bandwidth graph should look like this:

Confirming Our Math

I spent a lot of time making sure I got the math right and the query, but just to check, here’s a current graph from pfSense:

This maxes out at 500 megabits per second. Now let’s check the same time period in Grafana:

If we convert 500 megabits to megabytes by dividing by 8, we get 62.5. So…success! The math checks out! This also tells me that my cable provider upgraded my from 400 megabit package to 500 megabit.

Conclusion

You should be able to follow my previous guide for CPU statistics. One thing you may notice is that there are no memory statistics for your pfSense host. This is a bug that should be fixed at some point, but I’m on the latest version and it still hasn’t been fixed. I’ve yet to find a decent set of steps to fix it, but if I do, or it becomes fixed with a patch, I’ll update this post! Until next time…when we check out FreeNAS.