Automatic Volume Recycling
By default, once Bacula starts writing a Volume, it may append to the volume, but it will not overwrite the existing data thus destroying it. However when Bacula recycles a Volume, the Volume becomes available for being reused, and Bacula can at some later time overwrite the previous contents of that Volume. At that point all previous data on that Volume will be lost. If the Volume is a tape, the tape will be rewritten from the beginning. If the Volume is a disk file, the file will be truncated before being rewritten.
You may not want Bacula to automatically recycle (reuse) Volumes. Doing so may require a large number of Volumes though. However, it is also possible to manually recycle Volumes so that they may be reused. For more on manual recycling, see the section entitled Manually Recycling Volumes below in this chapter.
Most people prefer to have a Pool of Volumes that are used for daily backups and recycled once a week, another Pool of Volumes that are used for Full backups once a week and recycled monthly, and finally a Pool of Volumes that are used once a month and recycled after a year or two. With a scheme like this, the number of Volumes in your pool or pools remains constant.
By properly defining your Volume Pools with appropriate Retention periods, Bacula can manage the recycling (such as defined above) automatically.
Automatic recycling of Volumes is controlled by four records in the Pool resource definition in the Director's configuration file. These four records are:
- AutoPrune = yes
- VolumeRetention = <time>
- Recycle = yes
- RecyclePool = <APool>
- ScratchPool = <APool>
The above first three directives are all you need assuming that you fill each of your Volumes then wait the Volume Retention period before reusing them, providing there is some non-pruned Jobs or Files on the Volume. Recycle Pool and Scratch Pool directives are not required. If not defined, Bacula will recycle Volumes and keep them in the current Pool.
If you want Bacula to stop using a Volume and recycle it before it is full, you will need to use one or more additional directives such as:
- Use Volume Once = yes
- Volume Use Duration = ttt
- Maximum Volume Jobs = nnn
- Maximum Volume Bytes = mmm
Please see below and the Basic Volume Management chapter of this manual for more complete examples.
Automatic recycling of Volumes is performed by Bacula only when it wants a new Volume and no appendable Volumes are available in the Pool. It will then search the Pool for any Volumes with the Recycle flag set and the Volume Status is Purged. At that point, it will choose the oldest purged volume and recycle it.
If there are no volumes with status Purged, then the recycling occurs in two steps:
The first is that the Catalog for a Volume must be pruned of all Jobs (i.e. Purged) and Files contained on that Volume.
The second step is the actual recycling of the Volume. Only Volumes marked Full or Used will be considered for pruning. The Volume will be purged, all Jobs and Files associated to this Volume are pruned from Catalog, if the VolumeRetention period has expired. When a Volume is marked as Purged, it means that no Catalog records for Jobs and Files reference that Volume, and the Volume can be recycled and reused. Please note a Volume can be reused even though the Volume Retention period has not expired if the Jobs and Files associated to the Volume had been already pruned. Until recycling actually occurs, the Volume data remains intact. If no Volumes can be found for recycling for any of the reasons stated above, Bacula will request operator intervention (i.e. it will ask you to label a new volume).
A key point mentioned above, that can be a source of frustration, is that Bacula will only recycle purged Volumes if there is no other appendable Volume available, otherwise, it will always write to an appendable Volume before recycling even if there are Volumes marked as Purged. This preserves your data as long as possible. So, if you wish to “force” Bacula to use a purged Volume, you must first ensure that no other Volume in the Pool is marked Append. If necessary, you can manually set a volume to Full. The reason for this is that Bacula wants to preserve the data on your old Volumes (even though purged from the catalog) as long as absolutely possible before overwriting it. There are also a number of directives such as Volume Use Duration that will automatically mark a volume as Used and thus no longer appendable.
Automatic Pruning
As Bacula writes files to a Volume, it keeps a list of files, jobs, and volumes in a database called the catalog. Among other things, the database helps Bacula to decide which files to back up in an incremental or differential backup, and helps you locate files on past backups when you want to restore something. However, the catalog will grow larger and larger as time goes on, and eventually it can become unacceptably large.
Bacula's process for removing entries from the catalog is called Pruning. The default is Automatic Pruning, which means that once a Job record reaches a certain age (e.g. 30 days old) and a pruning occurs, it will be removed from the catalog. Note that Job records that are required for current restore won't be removed automatically, and File records are needed for VirtualFull and Accurate backups. Once a job has been pruned, you can still restore it from the backup Volume, provided that the Volume has not been recycled, but one additional step is required: scanning the volume with bscan. The alternative to Automatic Pruning is Manual Pruning, in which you explicitly tell Bacula to erase the catalog entries for a volume. You'd usually do this when you want to reuse a Bacula volume, because there's no point in keeping a list of files that USED TO BE on a volume. Or, if the catalog is starting to get too big, you could prune the oldest jobs to save space. Manual pruning is done with the prune command in the Bacula Enterprise Console manual (thanks to Bryce Denney for the above explanation).
Pruning Directives
There are three pruning durations. All apply to catalog database Jobs and Files records and not to the actual data in a Volume. The pruning (or retention) durations are for: Volumes (Jobs and Files records in the Volume), Jobs (Job records), and Files (File records). The durations inter-depend a bit because if Bacula prunes a Volume, it automatically removes all the Job records, and, consequently, all the File records. Also when a Job record is pruned, all the File records for that Job are also pruned (deleted) from the catalog.Having the File records in the database means that you can examine all the files backed up for a particular Job. They take the most space in the catalog (probably 90-95% of the total). When the File records are pruned, the Job records can remain, and you can still examine what Jobs ran, but not the details of the Files backed up. In addition, without the File records, you cannot use the Console restore command to restore specific files.
When a Job record is pruned, the Volume (Media record) for that Job can still remain in the database, and if you do a list volumes, you will see the volume information, but the Job records (and its File records) will no longer be available.
In each case, pruning removes information about where older files are, but it also prevents the catalog from growing to be too large. You choose the retention periods in function of how many files you are backing up and the time periods you want to keep those records online, and the size of the database. You can always re-insert the records (with 98% of the original data) by using bscan to scan in a whole Volume or any part of the volume that you want.
By setting AutoPrune to yes you will permit Bacula to automatically prune Volumes in the Pool when a Job needs a Volume. Volume pruning means removing Jobs and Files records from the catalog. It does not shrink the size of the Volume or affect the Volume data until the Volume gets overwritten. When a Job requests a volume and there are no Volumes with Volume Status Append, Recycle or Purged available, Bacula will begin volume pruning. This means that all Jobs that are older than the VolumeRetention period will be pruned from every Volume that has Volume Status Full or Used and has Recycle set to yes. Pruning consists of deleting the corresponding Job, File, and JobMedia records from the catalog database. No change to the physical data on the Volume occurs during the pruning process. When all files are pruned from a Volume (i.e. no records in the catalog), the Volume will be marked as Purged implying that no Jobs remain on the volume. The Pool records that control the pruning are described below.
- AutoPrune = <yes|no>
- If AutoPrune is set to yes (default), Bacula will automatically apply the Volume retention period when running a Job and it needs a new Volume but no appendable volumes are available. At that point, Bacula will prune all Volumes that can be pruned (i.e. AutoPrune set) in an attempt to find a usable volume. If during the autoprune, all files are pruned from the Volume, it will be marked with VolStatus Purged. The default is yes. Note, that although the File and Job records may be pruned from the catalog, a Volume will be marked Purged (and hence ready for recycling) if the Volume status is Append, Full, Used, or Error. If the Volume has another status, such as Archive, Read-Only, Disabled, Busy, or Cleaning, the Volume status will not be changed to Purged.
- Volume Retention = <time-period-specification>
- The Volume Retention record defines the length of time that Bacula will guarantee that the Volume is not reused (i.e. recycled). As long as a Volume has the status Append, it will not be recycled even though the retention period may have expired, until the status is changed to some other status such as Full, Used, Purged, ...
The retention period when applied starts at the time of the last write to the Volume. For a Volume to recycled, the volume must not have a status of Append and the retention period must have expired. Even in that case, the Volume will not be recycled if any other appendable Volume is available for the Job to complete. As noted above, the Volume status will be marked Purged by Bacula prior to it being recycled.Note, when all the Job records that are on the Volume have been removed, the Volume will be marked Purged (i.e. it has no more valid Jobs stored on it), and the Volume may be recycled even if the Volume Retention period has not expired.
When this time period expires, and if AutoPrune is set to yes, and a new Volume is needed, but no appendable Volume is available, Bacula will prune (remove) Job records that are older than the specified Volume Retention period even if the Job or File retention has not been reached. Normally this should not happen since the Volume Retention period should always be set greater than the Job Retention period, which should be greater than the File Retention period.
The Volume Retention period takes precedence over any Job Retention period you have specified in the Client resource. It should also be noted, that the Volume Retention period is obtained by reading the Catalog Database Media record rather than the Pool resource record. This means that if you change the VolumeRetention in the Pool resource record (in bacula-dir.conf), you must ensure that the corresponding change is made in the catalog by using the update pool command. Doing so will insure that any new Volumes will be created with the changed Volume Retention period. Any existing Volumes will have their own copy of the Volume Retention period that can only be changed on a Volume by Volume basis using the update volume command.
When all Job catalog entries are removed from the volume, its VolStatus is set to Purged. The files remain physically on the Volume until the volume is overwritten.
Retention periods are specified in seconds, minutes, hours, days, weeks, months, quarters, or years on the record. See the Configuration chapter of this manual for additional details of time specification.
The default Volume Retention period is 1 year.
- Recycle = <yes|no>
- This statement tells Bacula whether or not the particular Volume can be recycled (i.e. rewritten). If Recycle is set to no (the default), then even if Bacula prunes all the Jobs on the volume and it is marked Purged, it will not consider the volume for recycling. If Recycle is set to yes and all Jobs have been pruned, the volume status will be set to Purged and the volume may then be reused when another volume is needed. If the volume is reused, it is relabeled with the same Volume Name, however all previous data will be lost.
It is also possible to “force” pruning of all Volumes in the Pool associated with a Job by adding Prune Files = yes to the Job resource.
Recycling Algorithm
When a Job needs a Volume and no appendable, recycled or purged is available, Bacula starts the recycling algorithm pruning Jobs and Files for the oldest Volume in the pool used by the job. After the Volume is pruned, and if the Recycle flag is on (Recycle = yes) for that Volume, Bacula will relabel it and write new data on it.
As mentioned above, there are two key points for getting a Volume to be recycled. First, the Volume must no longer be marked Append (there are a number of directives to automatically make this change), and second since the last write on the Volume, one or more of the Retention periods must have expired so that there are no more catalog backup job records that reference that Volume. Once both those conditions are satisfied, the volume can be marked Purged and hence recycled.
The full algorithm that Bacula uses when it needs a new Volume is:
The algorithm described below assumes that AutoPrune is enabled, that Recycling is turned on, and that you have defined appropriate Retention periods, or used the defaults for all these items.
- If the request is for an Autochanger device, look only for Volumes in the Autochanger (i.e. with InChanger set and that have the correct Storage device).
- Search the Pool for a Volume with VolStatus=Append (if there is more than one, the Volume with the oldest date last written is chosen. If two have the same date then the one with the lowest MediaId is chosen).
- Search the Pool for a Volume with VolStatus=Recycle and the InChanger flag is set true (if there is more than one, the Volume with the oldest date last written is chosen. If two have the same date then the one with the lowest MediaId is chosen).
- Search the Pool for a Volume with VolStatus=Purged and try recycling any purged Volumes.
- Prune volumes applying Volume retention period (Volumes with VolStatus Full or Used are pruned). Note, when all the File and Job records are pruned from a Volume, the Volume will be marked Purged and this may be prior to the expiration of the Volume retention period.
- Prune the oldest Volume if RecycleOldestVolume = yes (the Volume with the oldest LastWritten date and VolStatus equal to Full or Used is chosen). This record ensures that all retention periods are properly respected.
- Purge the oldest Volume if PurgeOldestVolume=yes (the Volume with the oldest LastWritten date and VolStatus equal to Full or Used is chosen). We strongly recommend against the use of PurgeOldestVolume as it can quite easily lead to loss of current backup data.
- Search for a Volume in the Scratch Pool (the default Scratch pool or another Scratch Pool defined for the pool) and if found move it to the current Pool for the Job and use it. Note, when the Scratch Volume is moved into the current Pool, the basic Pool defaults are applied as if it is a newly labeled Volume (equivalent to an update volume from pool command).
- If we were looking for Volumes in the Autochanger, go back to step 2 above, but this time, look for any Volume whether or not it is in the Autochanger.
- Attempt to create a new Volume if automatic labeling is enabled and the maximum number of Volumes is not reached.
- Give up and ask operator.
The above occurs when Bacula has finished writing a Volume or when no Volume is present in the drive.
On the other hand, if you have inserted a different Volume after the last job, and Bacula recognizes the Volume as valid, the Storage daemon will request authorization from the Director to use this Volume. In this case, if you have set Recycle Current Volume = yes = nd the Volume is marked as Used or Full, Bacula will prune the volume and if all jobs were removed during the pruning (respecting the retention periods), the Volume will be recycled and re-used.
If you want to strictly observe the Volume Retention Period under all circumstances - i.e. you have a Volume with critical data, you must set Recycle = no. However, in doing so, you must manually do the recycling of the Volume for it to be used again.
The recycling algorithm in this case is:
- If the VolStatus is Append or Recycle is set, the volume will be used.
- If Recycle Current Volume is set and the volume is marked Full or Used, Bacula will prune the volume (applying the retention period). If all Jobs are pruned from the volume, it will be recycled.
This permits users to manually change the Volume every day and load volumes in an order different from what is in the catalog, and if the volume does not contain a current copy of your backup data, it will be used.
A few points to keep in mind:
- If a pool doesn't have maximum volumes defined then Bacula will prefer to demand new volumes over forcibly purging older volumes.
- If volumes become free through pruning, then they get marked as Purged and are immediately available for recycling - these will be used in preference to creating new volumes.
- If the Job, File, and Volume retention periods are different, then it's common to see a volume with no files or jobs listed in the database, but which is still not marked as Purged, if Autoprune is set to no in the pool resource.
Recycle Status
Each Volume inherits the Recycle status (yes or no) from the Pool resource record when the Media record is created (normally when the Volume is labeled). This Recycle status is stored in the Media record of the Catalog. Using the Console program, you may subsequently change the Recycle status for each Volume. For example in the following output from list volumes:
+----------+-------+--------+---------+------------+--------+-----+ | VolumeNa | Media | VolSta | VolByte | LastWritte | VolRet | Rec | +----------+-------+--------+---------+------------+--------+-----+ | File0001 | File | Full | 4190055 | 2002-05-25 | 14400 | 1 | | File0002 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 | | File0003 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 | | File0004 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 | | File0005 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 | | File0006 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 | | File0007 | File | Purged | 1896466 | 2002-05-26 | 14400 | 1 | +----------+-------+--------+---------+------------+--------+-----+
all the volumes are marked as recyclable, and the last Volume, File0007 has been purged, so it may be immediately recycled. The other volumes are all marked recyclable and when their Volume Retention period (14400 seconds or four hours) expires, they will be eligible for pruning, and possibly recycling. Even though Volume File0007 has been purged, all the data on the Volume is still recoverable. A purged Volume simply means that there are no entries in the Catalog. Even if the Volume Status is changed to Recycle, the data on the Volume will be recoverable. The data is lost only when the Volume is re-labeled and re-written.
To modify Volume File0001 so that it cannot be recycled, you use the update volume pool=File command in the console program, or simply update and Bacula will prompt you for the information.
+----------+------+-------+---------+-------------+-------+-----+ | VolumeNa | Media| VolSta| VolByte | LastWritten | VolRet| Rec | +----------+------+-------+---------+-------------+-------+-----+ | File0001 | File | Full | 4190055 | 2002-05-25 | 14400 | 0 | | File0002 | File | Full | 1897236 | 2002-05-26 | 14400 | 1 | | File0003 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 | | File0004 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 | | File0005 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 | | File0006 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 | | File0007 | File | Purged| 1896466 | 2002-05-26 | 14400 | 1 | +----------+------+-------+---------+-------------+-------+-----+
In this case, File0001 will never be automatically recycled. The same effect can be achieved by setting the Volume Status to Read-Only.
As you have noted, the Volume Status (VolStatus) column in the catalog database contains the current status of the Volume, which is normally maintained automatically by Bacula. To give you an idea of some of the values it can take during the life cycle of a Volume, here is a picture of the process.
A typical volume life cycle is like this:
because job count or size limit exceeded
Append ------------------------------------> Used/Full
^ |
| First Job writes to Retention time passed |
| the volume and recycling takes |
| place |
| v
Recycled <-------------------------------------- Purged
Volume is selected for reuse
Making Bacula Use a Single Tape
Most people will want Bacula to fill a tape and when it is full, a new tape will be mounted, and so on. However, as an extreme example, it is possible for Bacula to write on a single tape, and every night to rewrite it. To get this to work, you must do two things: first, set the VolumeRetention to less than your save period (one day), and the second item is to make Bacula mark the tape as full after using it once. This is done using UseVolumeOnce = yes. If this latter record is not used and the tape is not full after the first time it is written, Bacula will simply append to the tape and eventually request another volume. Using the tape only once, forces the tape to be marked Full after each use, and the next time Bacula runs, it will recycle the tape.
An example Pool resource that does this is:
Pool {
Name = DDS-4
Use Volume Once = yes
Pool Type = Backup
AutoPrune = yes
VolumeRetention = 12h # expire after 12 hours
Recycle = yes
}
Daily, Weekly, Monthly Tape Usage Example
This example is meant to show you how one could define a fixed set of volumes that Bacula will rotate through on a regular schedule. There are an infinite number of such schemes, all of which have various advantages and disadvantages. Note: these volumes may either be tape volumes or disk volumes.
We start with the following assumptions:
- A single tape has more than enough capacity to do a full save.
- There are ten tapes that are used on a daily basis for incremental backups. They are prelabeled Daily1 ... Daily10.
- There are four tapes that are used on a weekly basis for full backups. They are labeled Week1 ... Week4.
- There are 12 tapes that are used on a monthly basis for full backups. They are numbered Month1 ... Month12
- A full backup is done every Saturday evening (tape inserted Friday evening before leaving work).
- No backups are done over the weekend (this is easy to change).
- The first Friday of each month, a Monthly tape is used for the Full backup.
- Incremental backups are done Monday - Friday (actually Tue-Fri mornings).
We start the system by doing a Full save to one of the weekly volumes or one of the monthly volumes. The next morning, we remove the tape and insert a Daily tape. Friday evening, we remove the Daily tape and insert the next tape in the Weekly series. Monday, we remove the Weekly tape and re-insert the Daily tape. On the first Friday of the next month, we insert the next Monthly tape in the series rather than a Weekly tape, then continue. When a Daily tape finally fills up, Bacula will request the next one in the series, and the next day when you notice the email message, you will mount it and Bacula will finish the unfinished incremental backup.
What does this give? Well, at any point, you will have the last complete Full save plus several Incremental saves. For any given file you want to recover (or your whole system), you will have a copy of that file every day for at least the last 14 days. For older versions, you will have at least three and probably four Friday full saves of that file, and going back further, you will have a copy of that file made on the beginning of the month for at least a year.
So you have copies of any file (or your whole system) for at least a year, but as you go back in time, the time between copies increases from daily to weekly to monthly.
What would the Bacula configuration look like to implement such a scheme?
Schedule {
Name = "NightlySave"
Run = Level=Full Pool=Monthly 1st sat at 03:05
Run = Level=Full Pool=Weekly 2nd-5th sat at 03:05
Run = Level=Incremental Pool=Daily tue-fri at 03:05
}
Job {
Name = "NightlySave"
Type = Backup
Level = Full
Client = LocalMachine
FileSet = "File Set"
Messages = Standard
Storage = DDS-4
Pool = Daily
Schedule = "NightlySave"
}
# Definition of file storage device
Storage {
Name = DDS-4
Address = localhost
SDPort = 9103
Password = XXXXXXXXXXXXX
Device = FileStorage
Media Type = 8mm
}
FileSet {
Name = "File Set"
Include {
Options { signature=MD5 }
File = fffffffffffffffff
}
Exclude { File=*.o }
}
Pool {
Name = Daily
Pool Type = Backup
AutoPrune = yes
VolumeRetention = 10d # recycle in 10 days
Maximum Volumes = 10
Recycle = yes
}
Pool {
Name = Weekly
Use Volume Once = yes
Pool Type = Backup
AutoPrune = yes
VolumeRetention = 30d # recycle in 30 days (default)
Recycle = yes
}
Pool {
Name = Monthly
Use Volume Once = yes
Pool Type = Backup
AutoPrune = yes
VolumeRetention = 365d # recycle in 1 year
Recycle = yes
}
Automatic Pruning and Recycling Example
Perhaps the best way to understand the various resource records that come into play during automatic pruning and recycling is to run a Job that goes through the whole cycle. If you add the following resources to your Director's configuration file:
Schedule {
Name = "30 minute cycle"
Run = Level=Full Pool=File Messages=Standard Storage=File
hourly at 0:05
Run = Level=Full Pool=File Messages=Standard Storage=File
hourly at 0:35
}
Job {
Name = "Filetest"
Type = Backup
Level = Full
Client=XXXXXXXXXX
FileSet="Test Files"
Messages = Standard
Storage = File
Pool = File
Schedule = "30 minute cycle"
}
# Definition of file storage device
Storage {
Name = File
Address = XXXXXXXXXXX
SDPort = 9103
Password = XXXXXXXXXXXXX
Device = FileStorage
Media Type = File
}
FileSet {
Name = "File Set"
Include {
Options { signature=MD5 }
File = fffffffffffffffff
}
Exclude { File=*.o }
}
Pool {
Name = File
Use Volume Once = yes
Pool Type = Backup
LabelFormat = "File"
AutoPrune = yes
VolumeRetention = 4h
Maximum Volumes = 12
Recycle = yes
}
Where you will need to replace the ffffffffff's by the appropriate files to be saved for your configuration. For the FileSet Include, choose a directory that has one or two megabytes maximum since there will probably be approximately eight copies of the directory that Bacula will cycle through.
In addition, you will need to add the following to your Storage daemon's configuration file:
Device {
Name = FileStorage
Media Type = File
Archive Device = /tmp
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
}
With the above resources, Bacula will start a Job every half hour that saves a copy of the directory you chose to /tmp/File0001 ... /tmp/File0012. After 4 hours, Bacula will start recycling the backup Volumes (/tmp/File0001 ...). You should see this happening in the output produced. Bacula will automatically create the Volumes (Files) the first time it uses them.
To turn it off, either delete all the resources you've added, or simply comment out the Schedule record in the Job resource.
Manually Recycling Volumes
Although automatic recycling of Volumes is implemented in version 1.20 and later (see the Automatic Recycling of Volumes chapter of this manual), you may want to manually force reuse (recycling) of a Volume.
Assuming that you want to keep the Volume name, but you simply want to write new data on the tape, the steps to take are:
- Use the update volume command in the Console to ensure that the Recycle field is set to 1
- Use the purge jobs volume command in the Console to mark the Volume as Purged. Check by using list volumes.
Once the Volume is marked Purged, it will be recycled the next time a Volume is needed.
If you wish to reuse the volume by giving it a new name, follow the following steps:
- Use the purge jobs volume command in the Console to mark the Volume as Purged. Check by using list volumes.
- Use the Console relabel command to relabel the Volume.
Please note that the relabel command applies only to tape Volumes.
For Bacula versions prior to 1.30 or to manually relabel the Volume, use the instructions below:
- Use the delete volume command in the Console to delete the Volume from the Catalog.
- If a different tape is mounted, use the unmount command, remove the tape, and insert the tape to be renamed.
- Write an EOF mark in the tape using the following commands:
mt -f /dev/nst0 rewind mt -f /dev/nst0 weof
where you replace /dev/nst0 with the appropriate device name on your system.
- Use the label command to write a new label to the tape and to enter it in the catalog.
Please be aware that the delete command can be dangerous. Once it is done, to recover the File records, you must either restore your database as it was before the delete command, or use the bscan utility program to scan the tape and recreate the database entries.