Changes - j0ke.net Open Build Service

@@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: BaculaFS -Version: 0.1.0 +Version: 0.1.7 Summary: Bacula Filesystem in USErspace Home-page: http://code.google.com/p/baculafs Author: Avi Rozen @@ -10,64 +10,67 @@ BaculaFS ======== - BaculaFS_ - Exposes the Bacula_ catalog and storage as a Filesystem in + BaculaFS_ - Exposes the Bacula_ [*]_ catalog and storage as a Filesystem in USErspace (FUSE_). .. _BaculaFS: http://code.google.com/p/baculafs .. _Bacula: http://www.bacula.org .. _FUSE: http://fuse.sourceforge.net/ - Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> + Copyright |(C)| 2009, 2010 Avi Rozen <avi.rozen@gmail.com> - INTRODUCTION + .. contents:: + + Introduction ------------ - **BaculaFS** is a tool, independent of Bacula, that represents the - Bacula catalog and backup storage media as a read-only filesystem in - userspace. + **BaculaFS** is a tool, developed independently of Bacula, that + represents the Bacula catalog and backup storage media as a read-only + filesystem in userspace. **BaculaFS** is specifically designed to cater for the following use-cases: - maintaining a remote snapshot of the files in the backup storage - using `rsync`_ + using `rsync`_ or `duplicity`_ - auditing the contents of backup jobs, without resorting to SQL - queries + queries - comparing backup jobs (using several mount points) - Note that **BaculaFS** is a maintenance tool - it's operation may + Note that **BaculaFS** is a maintenance tool - its operation may interfere with the normal operation of a live Bacula setup (see the - LIMITATIONS_ section below). + Limitations_ section below). .. _rsync: http://samba.anu.edu.au/rsync/ + .. _duplicity: http://www.nongnu.org/duplicity/ - REQUIREMENTS + Requirements ------------ **BaculaFS** has been tested with the following set of required software packages: - + Bacula 3.0.2 with one of the following database backends: - - * SQLite_ 3.6.21 - * MySQL_ 5.4.1 - * PostgreSQL_ 8.3.9 + + Bacula 5.0.2 with one of the following database backends: + * SQLite_ 3.7.3 + * MySQL_ 5.1.49 + * PostgreSQL_ 8.4.5 + + FUSE_ components: - * Python FUSE 0.2 - * FUSE library 2.8.1 - * fusermount 2.8.1 - * FUSE kernel interface 7.12 - - + Python_ 2.5.4, with the following additional libraries: - - * MySQLdb_ 1.2.2 - * psycopg2_ 2.0.13 - * pexpect_ 2.3 - + * Python FUSE 0.2.1 + * FUSE library 2.8.4 + * fusermount 2.8.4 + * FUSE kernel interface 7.12 + + + Python_ 2.6.6, with the following additional libraries: + + * MySQLdb_ 1.2.2 + * psycopg2_ 2.2.1 + * pexpect_ 2.3 + + attr_ extended attributes utilities 2.4.44 **BaculaFS** requires the following in order to function: @@ -75,7 +78,7 @@ + access to the database that's used to store the Bacula catalog + access to the Bacula storage device + access to the Bacula ``bextract`` utility (bundled with the Bacula - storage daemon installation) + storage daemon installation) .. _SQLite: http://www.sqlite.org/ .. _MySQL: http://www.mysql.com/ @@ -88,7 +91,7 @@ .. _attr: http://savannah.nongnu.org/projects/attr - INSTALLATION + Installation ------------ Extract the source code archive to a temporary directory, ``cd`` to @@ -96,117 +99,136 @@ :: - python setup.py install + python setup.py install - USAGE + Usage ----- :: - baculafs [mountpoint] [options] - - Options: - --version show program's version number and exit - -h, --help show this help message and exit - -o opt,[opt...] mount options - -o driver=mysql|postgresql|sqlite|sqlite3 - database driver [default: sqlite3] - -o host=HOST database server address [default: localhost] - -o port=PORT database server port - -o database=PATH database name [default: bacula] - -o username=USERNAME database user name [default: bacula] - -o password=PASSWORD database password - -o conf=PATH storage daemon configuration file [default: - /etc/bacula/bacula-sd.conf] - -o client=CLIENT file daemon name - -o fileset=FILESET backup fileset - -o device=DEVICE storage device name [default: FileStorage] - -o datetime='YYYY-MM-DD hh:mm:ss' - snapshot date/time [default: now] - -o recent_job select contents of most recent job only [default: - False] - -o joblist='JOBID1 JOBID2 ...' - select contents of specified list of jobs - -o cleanup clean cache directory upon umount [default: False] - -o move_root make absolute path symlinks point to path under - mount point [default: False] - -o prefetch_attrs read and parse attributes for all files upon - filesystem initialization [default: False] - -o prefetch_symlinks extract all symbolic links upon filesystem - initialization (implies prefetch_attrs) [default: - False] - -o prefetch_regex=REGEX - extract all objects that match REGEX upon - filesystem initialization (implies prefetch_attrs) - -o prefetch_recent extract contents of most recent non-full job upon - filesystem initialization (implies - prefetch_symlinks) [default: False] - -o prefetch_diff=PATH extract files that do not match files at PATH - (hint: speeds up rsync; implies prefetch_symlinks) - -o prefetch_everything - extract everything upon filesystem initialization - (complete restore to cache) [default: False] - -o user_cache_path=PATH - user specified cache path (hint: combine this with - one of the prefetch options) [default: none] - -o logging=debug|info|warning|critical|error - logging level [default: warning] - -o syslog log to both syslog and console [default: False] - - FUSE options: - -d -o debug enable debug output (implies -f) - -f foreground operation - -s disable multi-threaded operation - - -o allow_other allow access to other users - -o allow_root allow access to root - -o nonempty allow mounts over non-empty file/dir - -o default_permissions enable permission checking by kernel - -o fsname=NAME set filesystem name - -o subtype=NAME set filesystem type - -o large_read issue large read requests (2.4 only) - -o max_read=N set maximum size of read requests - - -o hard_remove immediate removal (don't hide files) - -o use_ino let filesystem set inode numbers - -o readdir_ino try to fill in d_ino in readdir - -o direct_io use direct I/O - -o kernel_cache cache files in kernel - -o [no]auto_cache enable caching based on modification times (off) - -o umask=M set file permissions (octal) - -o uid=N set file owner - -o gid=N set file group - -o entry_timeout=T cache timeout for names (1.0s) - -o negative_timeout=T cache timeout for deleted names (0.0s) - -o attr_timeout=T cache timeout for attributes (1.0s) - -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout) - -o intr allow requests to be interrupted - -o intr_signal=NUM signal to send on interrupt (10) - -o modules=M1[:M2...] names of modules to push onto filesystem stack - - -o max_write=N set maximum size of write requests - -o max_readahead=N set maximum readahead - -o async_read perform reads asynchronously (default) - -o sync_read perform reads synchronously - -o atomic_o_trunc enable atomic open+truncate support - -o big_writes enable larger than 4kB writes - -o no_remote_lock disable remote file locking - - Module options: - - [subdir] - -o subdir=DIR prepend this directory to all paths (mandatory) - -o [no]rellinks transform absolute symlinks to relative - - [iconv] - -o from_code=CHARSET original encoding of file names (default: UTF-8) - -o to_code=CHARSET new encoding of the file names (default: UTF-8) - + baculafs [mountpoint] [options] + + Options: + --version show program's version number and exit + -h, --help show this help message and exit + -o opt,[opt...] mount options + -o driver=mysql|postgresql|sqlite3 + database driver [default: sqlite3] + -o host=HOST database server address [default: localhost] + -o port=PORT database server port + -o database=PATH database name [default: bacula] + -o username=USERNAME database user name [default: bacula] + -o password=PASSWORD database password (use '-o password= ' to get a + password prompt; if not provided, the password is + read from the DATABASE_PASSWORD environment + variable) + -o conf=PATH storage daemon configuration file [default: + /etc/bacula/bacula-sd.conf] + -o client=CLIENT file daemon name + -o fileset=FILESET backup fileset + -o device=DEVICE storage device name [default: FileStorage] + -o datetime='YYYY-MM-DD hh:mm:ss' + snapshot date/time [default: now] + -o recent_job select contents of most recent job only [default: + False] + -o joblist='JOBID1 JOBID2 ...' + select contents of specified list of jobs + -o cleanup clean cache directory upon umount [default: False] + -o move_root make absolute path symlinks point to path under + mount point [default: False] + -o prefetch_attrs read and parse attributes for all files upon + filesystem initialization [default: False] + -o prefetch_symlinks extract all symbolic links upon filesystem + initialization (implies prefetch_attrs) [default: + False] + -o prefetch_regex=REGEX + extract all objects that match REGEX upon + filesystem initialization (implies prefetch_attrs) + -o prefetch_recent extract contents of most recent non-full job upon + filesystem initialization (implies + prefetch_symlinks) [default: False] + -o prefetch_diff=PATH extract files that do not match files at PATH + (hint: speeds up rsync; implies prefetch_symlinks) + -o prefetch_difflist=DIFFLIST + extract files that do not match files in DIFFLIST + (list line format: 'Day Mon DD hh:mm:ss YYYY PATH'; + use '-' to read from standard input; hint: format + matches output of 'duplicity list-current-files -v0 + target_url'; implies prefetch_symlinks) + -o prefetch_list=LIST extract files that match files in LIST (list should + contains one absolute file path per line; use '-' + to read from standard input; implies + prefetch_symlinks) + -o prefetch_everything + extract everything upon filesystem initialization + (complete restore to cache) [default: False] + -o batch_list list files to be prefetched and exit [default: + False] + -o batch_bsr dump contnets of bsr file for extracting prefetched + files and exit [default: False] + -o batch_extract extract prefetched files to mount point and exit + [default: False] + -o user_cache_path=PATH + user specified cache path (hint: combine this with + one of the prefetch options) [default: none] + -o logging=debug|info|warning|critical|error + logging level [default: info] + -o syslog log to both syslog and console [default: False] + + FUSE options: + -d -o debug enable debug output (implies -f) + -f foreground operation + -s disable multi-threaded operation + + -o allow_other allow access to other users + -o allow_root allow access to root + -o nonempty allow mounts over non-empty file/dir + -o default_permissions enable permission checking by kernel + -o fsname=NAME set filesystem name + -o subtype=NAME set filesystem type + -o large_read issue large read requests (2.4 only) + -o max_read=N set maximum size of read requests + + -o hard_remove immediate removal (don't hide files) + -o use_ino let filesystem set inode numbers + -o readdir_ino try to fill in d_ino in readdir + -o direct_io use direct I/O + -o kernel_cache cache files in kernel + -o [no]auto_cache enable caching based on modification times (off) + -o umask=M set file permissions (octal) + -o uid=N set file owner + -o gid=N set file group + -o entry_timeout=T cache timeout for names (1.0s) + -o negative_timeout=T cache timeout for deleted names (0.0s) + -o attr_timeout=T cache timeout for attributes (1.0s) + -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout) + -o intr allow requests to be interrupted + -o intr_signal=NUM signal to send on interrupt (10) + -o modules=M1[:M2...] names of modules to push onto filesystem stack + + -o max_write=N set maximum size of write requests + -o max_readahead=N set maximum readahead + -o async_read perform reads asynchronously (default) + -o sync_read perform reads synchronously + -o atomic_o_trunc enable atomic open+truncate support + -o big_writes enable larger than 4kB writes + -o no_remote_lock disable remote file locking + + Module options: + + [subdir] + -o subdir=DIR prepend this directory to all paths (mandatory) + -o [no]rellinks transform absolute symlinks to relative + + [iconv] + -o from_code=CHARSET original encoding of file names (default: UTF-8) + -o to_code=CHARSET new encoding of the file names (default: UTF-8) + - OPERATION + Operation --------- - INITIALIZATION + Initialization ~~~~~~~~~~~~~~ **BaculaFS** starts by running several SQL queries against the Bacula @@ -223,7 +245,7 @@ At this point the filesystem is ready. - CACHE + Cache ~~~~~ Opening a file for reading causes **BaculaFS** to run ``bextract`` in @@ -239,9 +261,9 @@ For example: - use ``-o prefetch_attrs`` for storage space usage analysis - (e.g. with Baobab_) + (e.g. with Baobab_) - use ``-o prefetch_symlinks`` for any manual filesystem traversal - with command line or GUI tools (``find``, ``mc``, etc.) + with command line or GUI tools (``find``, ``mc``, etc.) - use ``-o prefetch_diff`` with ``rsync`` .. _Baobab: http://www.marzocca.net/linux/baobab/ @@ -250,7 +272,24 @@ filesystem, with ``-o cleanup``. It may also be reused between mount operations with ``-o user_cache_path``. - EXTENDED ATTRIBUTES + Batch Mode + ~~~~~~~~~~ + + **BaculaFS** may be used in *batch mode* with ``-o batch_extract`` in + order to extract files from the Bacula storage device and then exit + without mounting the filesystem. The mountpoint specified at the + command line is then treated as the destination directory for + extracted files. + + The list of files, to be extracted in batch mode, is determined by the + various cache prefetch options, and may be dumped with ``-o + batch_list``. + + The bootstrap file that is generated, in order to extract the files, + can also be dumped to standard output with ``-o batch_bsr``. + + + Extended Attributes ~~~~~~~~~~~~~~~~~~~ **BaculaFS** uses extended file attributes to expose Bacula specific @@ -259,35 +298,39 @@ :: - user.baculafs.FileIndex - user.baculafs.JobId - user.baculafs.LStat - user.baculafs.MD5 + user.baculafs.FileIndex + user.baculafs.JobId + user.baculafs.LStat + user.baculafs.MD5 + + Note that ``user.baculafs.MD5`` shows whatever digest Bacula was + configured to calculate for the file, be it MD5, SHA1, SHA256 or + SHA512. The root directory has several more attributes, that expose filesystem instance-specific information: :: - user.baculafs.cache_prefix - user.baculafs.client - user.baculafs.datetime - user.baculafs.fileset - user.baculafs.joblist + user.baculafs.cache_prefix + user.baculafs.client + user.baculafs.datetime + user.baculafs.fileset + user.baculafs.joblist and several more attributes for monitoring the file extraction process: - + :: - user.baculafs.bextract.failures - user.baculafs.bextract.path - user.baculafs.bextract.pending - user.baculafs.bextract.retries - user.baculafs.bextract.state - user.baculafs.bextract.volume + user.baculafs.bextract.failures + user.baculafs.bextract.path + user.baculafs.bextract.pending + user.baculafs.bextract.retries + user.baculafs.bextract.state + user.baculafs.bextract.volume - MISSING VOLUMES + Missing Volumes ~~~~~~~~~~~~~~~ If the storage device is a tape drive then it's possible that @@ -301,16 +344,98 @@ :: - attr -s baculafs.bextract.state -V run <mount-point> + attr -s baculafs.bextract.state -V run <mount-point> Please note that this feature has undergone only rudimentary testing. Expect breakage. - LIMITATIONS + Examples + ~~~~~~~~ + + Mount the most recent backup snapshot for Bacula client ``client-fd`` + and fileset ``client-fileset``; SQLite database backend (default): + + :: + + baculafs -o client=client-fd,fileset=client-fileset /path/to/mount/point + + Mount the contents of the specified list of backup jobs; MySQL database + backend; prompt for password and prefetch to cache all symbolic links: + + :: + + baculafs -o joblist='1001 1003',client=client-fd,fileset=client-fileset \ + -o driver=mysql,password=,prefetch_symlinks /path/to/mount/point + + Mount the contents of the most recent backup job only: + + :: + + baculafs -o recent_job,client=client-fd,fileset=client-fileset /path/to/mount/point + + Mount the contents of the fileset snapshot before the specified + date/time: + + :: + + baculafs -o datetime='2009-05-23 00:00:00' \ + -o client=client-fd,fileset=client-fileset /path/to/mount/point + + Allow other users to access filesystem, set logging level to ``debug`` + and stay in foreground, so that ``bextract`` messages may be examined; + assume a single Bacula fileset is defined for client ``client-fd``: + + :: + + baculafs -f -o allow_other,client=client-fd,logging=debug /path/to/mount/point + + Prefetch to cache files that do not exist, or whose modification time + or size differ from those at the specified directory (i.e. files that + would be sent to that directory using ``rsync``); and cleanup cache after + un-mounting: + + :: + + baculafs -o prefetch_diff=/path/to/rsync/target,cleanup \ + -o client=client-fd /path/to/mount/point + + Batch update a snapshot of the current files in backup (the filesystem + is *not* mounted): + + :: + + baculafs -o batch_extract,prefetch_diff=/path/to/snapshot,cleanup \ + -o client=client-fd /path/to/snapshot/ + + (this is not as accurate as mounting the filesystem, like in the + previous example, and then updating the snapshot with ``rsync`` - but + it is faster). + + + Limitations ----------- - LOCKING + Common Problems + ~~~~~~~~~~~~~~~ + + **BaculaFS** will abort if no backup job was run for the + client/fileset combination selected by the user. + + **BaculaFS** will abort if the target mount point is in use. Note, + however, that mount point availability is checked by FUSE only *after* + the (potentially lengthy) initialization of **BaculaFS**. + + Browsing a **BaculaFS** mount point with a file manager like Nautilus_ + is liable to be very slow. This is because the file manager reads data + from each file being listed, in order to determine its type, generate + a thumbnail etc. This, in turn, triggers **BaculaFS** to extract the + files, one at a time, from the Bacula storage device to its cache + directory. + + .. _Nautilus: http://live.gnome.org/Nautilus + + Locking ~~~~~~~ Access to the storage device by different instances of **BaculaFS** is @@ -323,28 +448,84 @@ should note: - the lock is *advisory*, meaning that it does not prevent the Bacula - storage daemon itself from accessing the storage device while in use - by **BaculaFS** + storage daemon itself from accessing the storage device while in use + by **BaculaFS** - depending on your setup, the lock may not work if the storage daemon - configuration file is accessed via NFS + configuration file is accessed via NFS + + + Unsupported Features + ~~~~~~~~~~~~~~~~~~~~ + + **BaculaFS** depends on ``bextract`` and thus inherits its + limitations: + + - **BaculaFS** can be used with Windows filesets, but it does not + reproduce any Windows specific file attributes + - Encrypted backup files are not supported + - **BaculaFS** must be started with enough permissions (typically as + ``root``) in order to allow ``bextract`` to extract files from the + Bacula storage + Changelog + --------- + **Version 0.1.7 (2010-12-30)** + + - fixed: compatibility issues with bacula v2.4.4 and FUSE 7.8 + - fixed: synthesize missing inode numbers with -o use_ino + - modified: decode value of user.baculafs.MD5 extended file attribute + - added: cache prefetch specified list of files + - added: changelog to README + - added: batch extract mode + + **Version 0.1.6 (2010-09-19)** + + - fixed cache prefetch by regex + + **Version 0.1.5 (2010-07-06)** + + - fixed: removed reference to obsolete db field Copy + - fixed: recent_job option with MySQL + - added: read database password from environment variable DATABASE_PASSWORD + - added: usage examples to README + + **Version 0.1.4 (2010-02-07)** + + - added: cache prefetch based on duplicity file listing + **Version 0.1.3 (2010-01-13)** - WINDOWS FILESETS - ~~~~~~~~~~~~~~~~ + - fixed: (again) prefetch restore of files split between volumes + - fixed: missing import sys + - fixed: spurious linebreaks in debug log - **BaculaFS** can be used with Windows backup filesets, but it does not - reproduce any Windows specific file attributes. This is because - ``bextract`` does not extract Windows specific file attributes on - Linux. + **Version 0.1.2 (2010-01-13)** - BUGS + - fixed: prefetch_recent when joblist contains a single non full job + - fixed: prefetch restore of files split between volumes + - added: copyright, trademark and license blurbs + + **Version 0.1.1 (2010-01-07)** + + - workaround: subtle extraction bug (fix forthcoming) + + **Version 0.1.1 (2010-01-06)** + + - initial public release + + Bugs ---- Please report problems via the **BaculaFS** issue tracking system: `<http://code.google.com/p/baculafs/issues/list>`_ - LICENSE + Credits + ------- + + **BaculaFS** contains SQL queries that were adapted from Bacula, + Copyright |(C)| 2000-2010 Free Software Foundation Europe e.V. + + License ------- **BaculaFS** is free software: you can redistribute it and/or modify @@ -361,11 +542,14 @@ along with this program. If not, see `<http://www.gnu.org/licenses/>`_. + .. [*] Bacula is a registered trademark of Kern Sibbald. + .. |(C)| unicode:: 0xA9 .. copyright sign Platform: Linux Classifier: Development Status :: 3 - Alpha +Classifier: Topic :: System :: Filesystems Classifier: Topic :: System :: Archiving :: Backup Classifier: Intended Audience :: System Administrators Classifier: Environment :: No Input/Output (Daemon)

@@ -16,4 +16,5 @@ baculafs/LogFile.py baculafs/SQL.py baculafs/__init__.py -baculafs/baculafs \ No newline at end of file +baculafs/baculafs +baculafs/junk.py \ No newline at end of file

@@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: BaculaFS -Version: 0.1.0 +Version: 0.1.7 Summary: Bacula Filesystem in USErspace Home-page: http://code.google.com/p/baculafs Author: Avi Rozen @@ -10,64 +10,67 @@ BaculaFS ======== - BaculaFS_ - Exposes the Bacula_ catalog and storage as a Filesystem in + BaculaFS_ - Exposes the Bacula_ [*]_ catalog and storage as a Filesystem in USErspace (FUSE_). .. _BaculaFS: http://code.google.com/p/baculafs .. _Bacula: http://www.bacula.org .. _FUSE: http://fuse.sourceforge.net/ - Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> + Copyright |(C)| 2009, 2010 Avi Rozen <avi.rozen@gmail.com> - INTRODUCTION + .. contents:: + + Introduction ------------ - **BaculaFS** is a tool, independent of Bacula, that represents the - Bacula catalog and backup storage media as a read-only filesystem in - userspace. + **BaculaFS** is a tool, developed independently of Bacula, that + represents the Bacula catalog and backup storage media as a read-only + filesystem in userspace. **BaculaFS** is specifically designed to cater for the following use-cases: - maintaining a remote snapshot of the files in the backup storage - using `rsync`_ + using `rsync`_ or `duplicity`_ - auditing the contents of backup jobs, without resorting to SQL - queries + queries - comparing backup jobs (using several mount points) - Note that **BaculaFS** is a maintenance tool - it's operation may + Note that **BaculaFS** is a maintenance tool - its operation may interfere with the normal operation of a live Bacula setup (see the - LIMITATIONS_ section below). + Limitations_ section below). .. _rsync: http://samba.anu.edu.au/rsync/ + .. _duplicity: http://www.nongnu.org/duplicity/ - REQUIREMENTS + Requirements ------------ **BaculaFS** has been tested with the following set of required software packages: - + Bacula 3.0.2 with one of the following database backends: - - * SQLite_ 3.6.21 - * MySQL_ 5.4.1 - * PostgreSQL_ 8.3.9 + + Bacula 5.0.2 with one of the following database backends: + * SQLite_ 3.7.3 + * MySQL_ 5.1.49 + * PostgreSQL_ 8.4.5 + + FUSE_ components: - * Python FUSE 0.2 - * FUSE library 2.8.1 - * fusermount 2.8.1 - * FUSE kernel interface 7.12 - - + Python_ 2.5.4, with the following additional libraries: - - * MySQLdb_ 1.2.2 - * psycopg2_ 2.0.13 - * pexpect_ 2.3 - + * Python FUSE 0.2.1 + * FUSE library 2.8.4 + * fusermount 2.8.4 + * FUSE kernel interface 7.12 + + + Python_ 2.6.6, with the following additional libraries: + + * MySQLdb_ 1.2.2 + * psycopg2_ 2.2.1 + * pexpect_ 2.3 + + attr_ extended attributes utilities 2.4.44 **BaculaFS** requires the following in order to function: @@ -75,7 +78,7 @@ + access to the database that's used to store the Bacula catalog + access to the Bacula storage device + access to the Bacula ``bextract`` utility (bundled with the Bacula - storage daemon installation) + storage daemon installation) .. _SQLite: http://www.sqlite.org/ .. _MySQL: http://www.mysql.com/ @@ -88,7 +91,7 @@ .. _attr: http://savannah.nongnu.org/projects/attr - INSTALLATION + Installation ------------ Extract the source code archive to a temporary directory, ``cd`` to @@ -96,117 +99,136 @@ :: - python setup.py install + python setup.py install - USAGE + Usage ----- :: - baculafs [mountpoint] [options] - - Options: - --version show program's version number and exit - -h, --help show this help message and exit - -o opt,[opt...] mount options - -o driver=mysql|postgresql|sqlite|sqlite3 - database driver [default: sqlite3] - -o host=HOST database server address [default: localhost] - -o port=PORT database server port - -o database=PATH database name [default: bacula] - -o username=USERNAME database user name [default: bacula] - -o password=PASSWORD database password - -o conf=PATH storage daemon configuration file [default: - /etc/bacula/bacula-sd.conf] - -o client=CLIENT file daemon name - -o fileset=FILESET backup fileset - -o device=DEVICE storage device name [default: FileStorage] - -o datetime='YYYY-MM-DD hh:mm:ss' - snapshot date/time [default: now] - -o recent_job select contents of most recent job only [default: - False] - -o joblist='JOBID1 JOBID2 ...' - select contents of specified list of jobs - -o cleanup clean cache directory upon umount [default: False] - -o move_root make absolute path symlinks point to path under - mount point [default: False] - -o prefetch_attrs read and parse attributes for all files upon - filesystem initialization [default: False] - -o prefetch_symlinks extract all symbolic links upon filesystem - initialization (implies prefetch_attrs) [default: - False] - -o prefetch_regex=REGEX - extract all objects that match REGEX upon - filesystem initialization (implies prefetch_attrs) - -o prefetch_recent extract contents of most recent non-full job upon - filesystem initialization (implies - prefetch_symlinks) [default: False] - -o prefetch_diff=PATH extract files that do not match files at PATH - (hint: speeds up rsync; implies prefetch_symlinks) - -o prefetch_everything - extract everything upon filesystem initialization - (complete restore to cache) [default: False] - -o user_cache_path=PATH - user specified cache path (hint: combine this with - one of the prefetch options) [default: none] - -o logging=debug|info|warning|critical|error - logging level [default: warning] - -o syslog log to both syslog and console [default: False] - - FUSE options: - -d -o debug enable debug output (implies -f) - -f foreground operation - -s disable multi-threaded operation - - -o allow_other allow access to other users - -o allow_root allow access to root - -o nonempty allow mounts over non-empty file/dir - -o default_permissions enable permission checking by kernel - -o fsname=NAME set filesystem name - -o subtype=NAME set filesystem type - -o large_read issue large read requests (2.4 only) - -o max_read=N set maximum size of read requests - - -o hard_remove immediate removal (don't hide files) - -o use_ino let filesystem set inode numbers - -o readdir_ino try to fill in d_ino in readdir - -o direct_io use direct I/O - -o kernel_cache cache files in kernel - -o [no]auto_cache enable caching based on modification times (off) - -o umask=M set file permissions (octal) - -o uid=N set file owner - -o gid=N set file group - -o entry_timeout=T cache timeout for names (1.0s) - -o negative_timeout=T cache timeout for deleted names (0.0s) - -o attr_timeout=T cache timeout for attributes (1.0s) - -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout) - -o intr allow requests to be interrupted - -o intr_signal=NUM signal to send on interrupt (10) - -o modules=M1[:M2...] names of modules to push onto filesystem stack - - -o max_write=N set maximum size of write requests - -o max_readahead=N set maximum readahead - -o async_read perform reads asynchronously (default) - -o sync_read perform reads synchronously - -o atomic_o_trunc enable atomic open+truncate support - -o big_writes enable larger than 4kB writes - -o no_remote_lock disable remote file locking - - Module options: - - [subdir] - -o subdir=DIR prepend this directory to all paths (mandatory) - -o [no]rellinks transform absolute symlinks to relative - - [iconv] - -o from_code=CHARSET original encoding of file names (default: UTF-8) - -o to_code=CHARSET new encoding of the file names (default: UTF-8) - + baculafs [mountpoint] [options] + + Options: + --version show program's version number and exit + -h, --help show this help message and exit + -o opt,[opt...] mount options + -o driver=mysql|postgresql|sqlite3 + database driver [default: sqlite3] + -o host=HOST database server address [default: localhost] + -o port=PORT database server port + -o database=PATH database name [default: bacula] + -o username=USERNAME database user name [default: bacula] + -o password=PASSWORD database password (use '-o password= ' to get a + password prompt; if not provided, the password is + read from the DATABASE_PASSWORD environment + variable) + -o conf=PATH storage daemon configuration file [default: + /etc/bacula/bacula-sd.conf] + -o client=CLIENT file daemon name + -o fileset=FILESET backup fileset + -o device=DEVICE storage device name [default: FileStorage] + -o datetime='YYYY-MM-DD hh:mm:ss' + snapshot date/time [default: now] + -o recent_job select contents of most recent job only [default: + False] + -o joblist='JOBID1 JOBID2 ...' + select contents of specified list of jobs + -o cleanup clean cache directory upon umount [default: False] + -o move_root make absolute path symlinks point to path under + mount point [default: False] + -o prefetch_attrs read and parse attributes for all files upon + filesystem initialization [default: False] + -o prefetch_symlinks extract all symbolic links upon filesystem + initialization (implies prefetch_attrs) [default: + False] + -o prefetch_regex=REGEX + extract all objects that match REGEX upon + filesystem initialization (implies prefetch_attrs) + -o prefetch_recent extract contents of most recent non-full job upon + filesystem initialization (implies + prefetch_symlinks) [default: False] + -o prefetch_diff=PATH extract files that do not match files at PATH + (hint: speeds up rsync; implies prefetch_symlinks) + -o prefetch_difflist=DIFFLIST + extract files that do not match files in DIFFLIST + (list line format: 'Day Mon DD hh:mm:ss YYYY PATH'; + use '-' to read from standard input; hint: format + matches output of 'duplicity list-current-files -v0 + target_url'; implies prefetch_symlinks) + -o prefetch_list=LIST extract files that match files in LIST (list should + contains one absolute file path per line; use '-' + to read from standard input; implies + prefetch_symlinks) + -o prefetch_everything + extract everything upon filesystem initialization + (complete restore to cache) [default: False] + -o batch_list list files to be prefetched and exit [default: + False] + -o batch_bsr dump contnets of bsr file for extracting prefetched + files and exit [default: False] + -o batch_extract extract prefetched files to mount point and exit + [default: False] + -o user_cache_path=PATH + user specified cache path (hint: combine this with + one of the prefetch options) [default: none] + -o logging=debug|info|warning|critical|error + logging level [default: info] + -o syslog log to both syslog and console [default: False] + + FUSE options: + -d -o debug enable debug output (implies -f) + -f foreground operation + -s disable multi-threaded operation + + -o allow_other allow access to other users + -o allow_root allow access to root + -o nonempty allow mounts over non-empty file/dir + -o default_permissions enable permission checking by kernel + -o fsname=NAME set filesystem name + -o subtype=NAME set filesystem type + -o large_read issue large read requests (2.4 only) + -o max_read=N set maximum size of read requests + + -o hard_remove immediate removal (don't hide files) + -o use_ino let filesystem set inode numbers + -o readdir_ino try to fill in d_ino in readdir + -o direct_io use direct I/O + -o kernel_cache cache files in kernel + -o [no]auto_cache enable caching based on modification times (off) + -o umask=M set file permissions (octal) + -o uid=N set file owner + -o gid=N set file group + -o entry_timeout=T cache timeout for names (1.0s) + -o negative_timeout=T cache timeout for deleted names (0.0s) + -o attr_timeout=T cache timeout for attributes (1.0s) + -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout) + -o intr allow requests to be interrupted + -o intr_signal=NUM signal to send on interrupt (10) + -o modules=M1[:M2...] names of modules to push onto filesystem stack + + -o max_write=N set maximum size of write requests + -o max_readahead=N set maximum readahead + -o async_read perform reads asynchronously (default) + -o sync_read perform reads synchronously + -o atomic_o_trunc enable atomic open+truncate support + -o big_writes enable larger than 4kB writes + -o no_remote_lock disable remote file locking + + Module options: + + [subdir] + -o subdir=DIR prepend this directory to all paths (mandatory) + -o [no]rellinks transform absolute symlinks to relative + + [iconv] + -o from_code=CHARSET original encoding of file names (default: UTF-8) + -o to_code=CHARSET new encoding of the file names (default: UTF-8) + - OPERATION + Operation --------- - INITIALIZATION + Initialization ~~~~~~~~~~~~~~ **BaculaFS** starts by running several SQL queries against the Bacula @@ -223,7 +245,7 @@ At this point the filesystem is ready. - CACHE + Cache ~~~~~ Opening a file for reading causes **BaculaFS** to run ``bextract`` in @@ -239,9 +261,9 @@ For example: - use ``-o prefetch_attrs`` for storage space usage analysis - (e.g. with Baobab_) + (e.g. with Baobab_) - use ``-o prefetch_symlinks`` for any manual filesystem traversal - with command line or GUI tools (``find``, ``mc``, etc.) + with command line or GUI tools (``find``, ``mc``, etc.) - use ``-o prefetch_diff`` with ``rsync`` .. _Baobab: http://www.marzocca.net/linux/baobab/ @@ -250,7 +272,24 @@ filesystem, with ``-o cleanup``. It may also be reused between mount operations with ``-o user_cache_path``. - EXTENDED ATTRIBUTES + Batch Mode + ~~~~~~~~~~ + + **BaculaFS** may be used in *batch mode* with ``-o batch_extract`` in + order to extract files from the Bacula storage device and then exit + without mounting the filesystem. The mountpoint specified at the + command line is then treated as the destination directory for + extracted files. + + The list of files, to be extracted in batch mode, is determined by the + various cache prefetch options, and may be dumped with ``-o + batch_list``. + + The bootstrap file that is generated, in order to extract the files, + can also be dumped to standard output with ``-o batch_bsr``. + + + Extended Attributes ~~~~~~~~~~~~~~~~~~~ **BaculaFS** uses extended file attributes to expose Bacula specific @@ -259,35 +298,39 @@ :: - user.baculafs.FileIndex - user.baculafs.JobId - user.baculafs.LStat - user.baculafs.MD5 + user.baculafs.FileIndex + user.baculafs.JobId + user.baculafs.LStat + user.baculafs.MD5 + + Note that ``user.baculafs.MD5`` shows whatever digest Bacula was + configured to calculate for the file, be it MD5, SHA1, SHA256 or + SHA512. The root directory has several more attributes, that expose filesystem instance-specific information: :: - user.baculafs.cache_prefix - user.baculafs.client - user.baculafs.datetime - user.baculafs.fileset - user.baculafs.joblist + user.baculafs.cache_prefix + user.baculafs.client + user.baculafs.datetime + user.baculafs.fileset + user.baculafs.joblist and several more attributes for monitoring the file extraction process: - + :: - user.baculafs.bextract.failures - user.baculafs.bextract.path - user.baculafs.bextract.pending - user.baculafs.bextract.retries - user.baculafs.bextract.state - user.baculafs.bextract.volume + user.baculafs.bextract.failures + user.baculafs.bextract.path + user.baculafs.bextract.pending + user.baculafs.bextract.retries + user.baculafs.bextract.state + user.baculafs.bextract.volume - MISSING VOLUMES + Missing Volumes ~~~~~~~~~~~~~~~ If the storage device is a tape drive then it's possible that @@ -301,16 +344,98 @@ :: - attr -s baculafs.bextract.state -V run <mount-point> + attr -s baculafs.bextract.state -V run <mount-point> Please note that this feature has undergone only rudimentary testing. Expect breakage. - LIMITATIONS + Examples + ~~~~~~~~ + + Mount the most recent backup snapshot for Bacula client ``client-fd`` + and fileset ``client-fileset``; SQLite database backend (default): + + :: + + baculafs -o client=client-fd,fileset=client-fileset /path/to/mount/point + + Mount the contents of the specified list of backup jobs; MySQL database + backend; prompt for password and prefetch to cache all symbolic links: + + :: + + baculafs -o joblist='1001 1003',client=client-fd,fileset=client-fileset \ + -o driver=mysql,password=,prefetch_symlinks /path/to/mount/point + + Mount the contents of the most recent backup job only: + + :: + + baculafs -o recent_job,client=client-fd,fileset=client-fileset /path/to/mount/point + + Mount the contents of the fileset snapshot before the specified + date/time: + + :: + + baculafs -o datetime='2009-05-23 00:00:00' \ + -o client=client-fd,fileset=client-fileset /path/to/mount/point + + Allow other users to access filesystem, set logging level to ``debug`` + and stay in foreground, so that ``bextract`` messages may be examined; + assume a single Bacula fileset is defined for client ``client-fd``: + + :: + + baculafs -f -o allow_other,client=client-fd,logging=debug /path/to/mount/point + + Prefetch to cache files that do not exist, or whose modification time + or size differ from those at the specified directory (i.e. files that + would be sent to that directory using ``rsync``); and cleanup cache after + un-mounting: + + :: + + baculafs -o prefetch_diff=/path/to/rsync/target,cleanup \ + -o client=client-fd /path/to/mount/point + + Batch update a snapshot of the current files in backup (the filesystem + is *not* mounted): + + :: + + baculafs -o batch_extract,prefetch_diff=/path/to/snapshot,cleanup \ + -o client=client-fd /path/to/snapshot/ + + (this is not as accurate as mounting the filesystem, like in the + previous example, and then updating the snapshot with ``rsync`` - but + it is faster). + + + Limitations ----------- - LOCKING + Common Problems + ~~~~~~~~~~~~~~~ + + **BaculaFS** will abort if no backup job was run for the + client/fileset combination selected by the user. + + **BaculaFS** will abort if the target mount point is in use. Note, + however, that mount point availability is checked by FUSE only *after* + the (potentially lengthy) initialization of **BaculaFS**. + + Browsing a **BaculaFS** mount point with a file manager like Nautilus_ + is liable to be very slow. This is because the file manager reads data + from each file being listed, in order to determine its type, generate + a thumbnail etc. This, in turn, triggers **BaculaFS** to extract the + files, one at a time, from the Bacula storage device to its cache + directory. + + .. _Nautilus: http://live.gnome.org/Nautilus + + Locking ~~~~~~~ Access to the storage device by different instances of **BaculaFS** is @@ -323,28 +448,84 @@ should note: - the lock is *advisory*, meaning that it does not prevent the Bacula - storage daemon itself from accessing the storage device while in use - by **BaculaFS** + storage daemon itself from accessing the storage device while in use + by **BaculaFS** - depending on your setup, the lock may not work if the storage daemon - configuration file is accessed via NFS + configuration file is accessed via NFS + + + Unsupported Features + ~~~~~~~~~~~~~~~~~~~~ + + **BaculaFS** depends on ``bextract`` and thus inherits its + limitations: + + - **BaculaFS** can be used with Windows filesets, but it does not + reproduce any Windows specific file attributes + - Encrypted backup files are not supported + - **BaculaFS** must be started with enough permissions (typically as + ``root``) in order to allow ``bextract`` to extract files from the + Bacula storage + Changelog + --------- + **Version 0.1.7 (2010-12-30)** + + - fixed: compatibility issues with bacula v2.4.4 and FUSE 7.8 + - fixed: synthesize missing inode numbers with -o use_ino + - modified: decode value of user.baculafs.MD5 extended file attribute + - added: cache prefetch specified list of files + - added: changelog to README + - added: batch extract mode + + **Version 0.1.6 (2010-09-19)** + + - fixed cache prefetch by regex + + **Version 0.1.5 (2010-07-06)** + + - fixed: removed reference to obsolete db field Copy + - fixed: recent_job option with MySQL + - added: read database password from environment variable DATABASE_PASSWORD + - added: usage examples to README + + **Version 0.1.4 (2010-02-07)** + + - added: cache prefetch based on duplicity file listing + **Version 0.1.3 (2010-01-13)** - WINDOWS FILESETS - ~~~~~~~~~~~~~~~~ + - fixed: (again) prefetch restore of files split between volumes + - fixed: missing import sys + - fixed: spurious linebreaks in debug log - **BaculaFS** can be used with Windows backup filesets, but it does not - reproduce any Windows specific file attributes. This is because - ``bextract`` does not extract Windows specific file attributes on - Linux. + **Version 0.1.2 (2010-01-13)** - BUGS + - fixed: prefetch_recent when joblist contains a single non full job + - fixed: prefetch restore of files split between volumes + - added: copyright, trademark and license blurbs + + **Version 0.1.1 (2010-01-07)** + + - workaround: subtle extraction bug (fix forthcoming) + + **Version 0.1.1 (2010-01-06)** + + - initial public release + + Bugs ---- Please report problems via the **BaculaFS** issue tracking system: `<http://code.google.com/p/baculafs/issues/list>`_ - LICENSE + Credits + ------- + + **BaculaFS** contains SQL queries that were adapted from Bacula, + Copyright |(C)| 2000-2010 Free Software Foundation Europe e.V. + + License ------- **BaculaFS** is free software: you can redistribute it and/or modify @@ -361,11 +542,14 @@ along with this program. If not, see `<http://www.gnu.org/licenses/>`_. + .. [*] Bacula is a registered trademark of Kern Sibbald. + .. |(C)| unicode:: 0xA9 .. copyright sign Platform: Linux Classifier: Development Status :: 3 - Alpha +Classifier: Topic :: System :: Filesystems Classifier: Topic :: System :: Archiving :: Backup Classifier: Intended Audience :: System Administrators Classifier: Environment :: No Input/Output (Daemon)

@@ -2,63 +2,66 @@ BaculaFS ======== -BaculaFS_ - Exposes the Bacula_ catalog and storage as a Filesystem in +BaculaFS_ - Exposes the Bacula_ [*]_ catalog and storage as a Filesystem in USErspace (FUSE_). .. _BaculaFS: http://code.google.com/p/baculafs .. _Bacula: http://www.bacula.org .. _FUSE: http://fuse.sourceforge.net/ -Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +Copyright |(C)| 2009, 2010 Avi Rozen <avi.rozen@gmail.com> -INTRODUCTION +.. contents:: + +Introduction ------------ -**BaculaFS** is a tool, independent of Bacula, that represents the -Bacula catalog and backup storage media as a read-only filesystem in -userspace. +**BaculaFS** is a tool, developed independently of Bacula, that +represents the Bacula catalog and backup storage media as a read-only +filesystem in userspace. **BaculaFS** is specifically designed to cater for the following use-cases: - maintaining a remote snapshot of the files in the backup storage - using `rsync`_ + using `rsync`_ or `duplicity`_ - auditing the contents of backup jobs, without resorting to SQL queries - comparing backup jobs (using several mount points) -Note that **BaculaFS** is a maintenance tool - it's operation may +Note that **BaculaFS** is a maintenance tool - its operation may interfere with the normal operation of a live Bacula setup (see the -LIMITATIONS_ section below). +Limitations_ section below). .. _rsync: http://samba.anu.edu.au/rsync/ +.. _duplicity: http://www.nongnu.org/duplicity/ -REQUIREMENTS +Requirements ------------ **BaculaFS** has been tested with the following set of required software packages: -+ Bacula 3.0.2 with one of the following database backends: ++ Bacula 5.0.2 with one of the following database backends: - * SQLite_ 3.6.21 - * MySQL_ 5.4.1 - * PostgreSQL_ 8.3.9 + * SQLite_ 3.7.3 + * MySQL_ 5.1.49 + * PostgreSQL_ 8.4.5 + FUSE_ components: - * Python FUSE 0.2 - * FUSE library 2.8.1 - * fusermount 2.8.1 + * Python FUSE 0.2.1 + * FUSE library 2.8.4 + * fusermount 2.8.4 * FUSE kernel interface 7.12 -+ Python_ 2.5.4, with the following additional libraries: ++ Python_ 2.6.6, with the following additional libraries: * MySQLdb_ 1.2.2 - * psycopg2_ 2.0.13 - * pexpect_ 2.3 + * psycopg2_ 2.2.1 + * pexpect_ 2.3 + attr_ extended attributes utilities 2.4.44 @@ -80,7 +83,7 @@ .. _attr: http://savannah.nongnu.org/projects/attr -INSTALLATION +Installation ------------ Extract the source code archive to a temporary directory, ``cd`` to @@ -90,7 +93,7 @@ python setup.py install -USAGE +Usage ----- :: @@ -101,13 +104,16 @@ --version show program's version number and exit -h, --help show this help message and exit -o opt,[opt...] mount options - -o driver=mysql|postgresql|sqlite|sqlite3 + -o driver=mysql|postgresql|sqlite3 database driver [default: sqlite3] -o host=HOST database server address [default: localhost] -o port=PORT database server port -o database=PATH database name [default: bacula] -o username=USERNAME database user name [default: bacula] - -o password=PASSWORD database password + -o password=PASSWORD database password (use '-o password= ' to get a + password prompt; if not provided, the password is + read from the DATABASE_PASSWORD environment + variable) -o conf=PATH storage daemon configuration file [default: /etc/bacula/bacula-sd.conf] -o client=CLIENT file daemon name @@ -135,14 +141,30 @@ prefetch_symlinks) [default: False] -o prefetch_diff=PATH extract files that do not match files at PATH (hint: speeds up rsync; implies prefetch_symlinks) + -o prefetch_difflist=DIFFLIST + extract files that do not match files in DIFFLIST + (list line format: 'Day Mon DD hh:mm:ss YYYY PATH'; + use '-' to read from standard input; hint: format + matches output of 'duplicity list-current-files -v0 + target_url'; implies prefetch_symlinks) + -o prefetch_list=LIST extract files that match files in LIST (list should + contains one absolute file path per line; use '-' + to read from standard input; implies + prefetch_symlinks) -o prefetch_everything extract everything upon filesystem initialization (complete restore to cache) [default: False] + -o batch_list list files to be prefetched and exit [default: + False] + -o batch_bsr dump contnets of bsr file for extracting prefetched + files and exit [default: False] + -o batch_extract extract prefetched files to mount point and exit + [default: False] -o user_cache_path=PATH user specified cache path (hint: combine this with one of the prefetch options) [default: none] -o logging=debug|info|warning|critical|error - logging level [default: warning] + logging level [default: info] -o syslog log to both syslog and console [default: False] FUSE options: @@ -195,10 +217,10 @@ -o to_code=CHARSET new encoding of the file names (default: UTF-8) -OPERATION +Operation --------- -INITIALIZATION +Initialization ~~~~~~~~~~~~~~ **BaculaFS** starts by running several SQL queries against the Bacula @@ -215,7 +237,7 @@ At this point the filesystem is ready. -CACHE +Cache ~~~~~ Opening a file for reading causes **BaculaFS** to run ``bextract`` in @@ -242,7 +264,24 @@ filesystem, with ``-o cleanup``. It may also be reused between mount operations with ``-o user_cache_path``. -EXTENDED ATTRIBUTES +Batch Mode +~~~~~~~~~~ + +**BaculaFS** may be used in *batch mode* with ``-o batch_extract`` in +order to extract files from the Bacula storage device and then exit +without mounting the filesystem. The mountpoint specified at the +command line is then treated as the destination directory for +extracted files. + +The list of files, to be extracted in batch mode, is determined by the +various cache prefetch options, and may be dumped with ``-o +batch_list``. + +The bootstrap file that is generated, in order to extract the files, +can also be dumped to standard output with ``-o batch_bsr``. + + +Extended Attributes ~~~~~~~~~~~~~~~~~~~ **BaculaFS** uses extended file attributes to expose Bacula specific @@ -256,6 +295,10 @@ user.baculafs.LStat user.baculafs.MD5 +Note that ``user.baculafs.MD5`` shows whatever digest Bacula was +configured to calculate for the file, be it MD5, SHA1, SHA256 or +SHA512. + The root directory has several more attributes, that expose filesystem instance-specific information: @@ -279,7 +322,7 @@ user.baculafs.bextract.state user.baculafs.bextract.volume -MISSING VOLUMES +Missing Volumes ~~~~~~~~~~~~~~~ If the storage device is a tape drive then it's possible that @@ -299,10 +342,92 @@ testing. Expect breakage. -LIMITATIONS +Examples +~~~~~~~~ + +Mount the most recent backup snapshot for Bacula client ``client-fd`` +and fileset ``client-fileset``; SQLite database backend (default): + +:: + + baculafs -o client=client-fd,fileset=client-fileset /path/to/mount/point + +Mount the contents of the specified list of backup jobs; MySQL database +backend; prompt for password and prefetch to cache all symbolic links: + +:: + + baculafs -o joblist='1001 1003',client=client-fd,fileset=client-fileset \ + -o driver=mysql,password=,prefetch_symlinks /path/to/mount/point + +Mount the contents of the most recent backup job only: + +:: + + baculafs -o recent_job,client=client-fd,fileset=client-fileset /path/to/mount/point + +Mount the contents of the fileset snapshot before the specified +date/time: + +:: + + baculafs -o datetime='2009-05-23 00:00:00' \ + -o client=client-fd,fileset=client-fileset /path/to/mount/point + +Allow other users to access filesystem, set logging level to ``debug`` +and stay in foreground, so that ``bextract`` messages may be examined; +assume a single Bacula fileset is defined for client ``client-fd``: + +:: + + baculafs -f -o allow_other,client=client-fd,logging=debug /path/to/mount/point + +Prefetch to cache files that do not exist, or whose modification time +or size differ from those at the specified directory (i.e. files that +would be sent to that directory using ``rsync``); and cleanup cache after +un-mounting: + +:: + + baculafs -o prefetch_diff=/path/to/rsync/target,cleanup \ + -o client=client-fd /path/to/mount/point + +Batch update a snapshot of the current files in backup (the filesystem +is *not* mounted): + +:: + + baculafs -o batch_extract,prefetch_diff=/path/to/snapshot,cleanup \ + -o client=client-fd /path/to/snapshot/ + +(this is not as accurate as mounting the filesystem, like in the +previous example, and then updating the snapshot with ``rsync`` - but +it is faster). + + +Limitations ----------- -LOCKING +Common Problems +~~~~~~~~~~~~~~~ + +**BaculaFS** will abort if no backup job was run for the +client/fileset combination selected by the user. + +**BaculaFS** will abort if the target mount point is in use. Note, +however, that mount point availability is checked by FUSE only *after* +the (potentially lengthy) initialization of **BaculaFS**. + +Browsing a **BaculaFS** mount point with a file manager like Nautilus_ +is liable to be very slow. This is because the file manager reads data +from each file being listed, in order to determine its type, generate +a thumbnail etc. This, in turn, triggers **BaculaFS** to extract the +files, one at a time, from the Bacula storage device to its cache +directory. + +.. _Nautilus: http://live.gnome.org/Nautilus + +Locking ~~~~~~~ Access to the storage device by different instances of **BaculaFS** is @@ -321,22 +446,78 @@ configuration file is accessed via NFS +Unsupported Features +~~~~~~~~~~~~~~~~~~~~ + +**BaculaFS** depends on ``bextract`` and thus inherits its +limitations: + +- **BaculaFS** can be used with Windows filesets, but it does not + reproduce any Windows specific file attributes +- Encrypted backup files are not supported +- **BaculaFS** must be started with enough permissions (typically as + ``root``) in order to allow ``bextract`` to extract files from the + Bacula storage + +Changelog +--------- +**Version 0.1.7 (2010-12-30)** + +- fixed: compatibility issues with bacula v2.4.4 and FUSE 7.8 +- fixed: synthesize missing inode numbers with -o use_ino +- modified: decode value of user.baculafs.MD5 extended file attribute +- added: cache prefetch specified list of files +- added: changelog to README +- added: batch extract mode -WINDOWS FILESETS -~~~~~~~~~~~~~~~~ +**Version 0.1.6 (2010-09-19)** -**BaculaFS** can be used with Windows backup filesets, but it does not -reproduce any Windows specific file attributes. This is because -``bextract`` does not extract Windows specific file attributes on -Linux. +- fixed cache prefetch by regex -BUGS +**Version 0.1.5 (2010-07-06)** + +- fixed: removed reference to obsolete db field Copy +- fixed: recent_job option with MySQL +- added: read database password from environment variable DATABASE_PASSWORD +- added: usage examples to README + +**Version 0.1.4 (2010-02-07)** + +- added: cache prefetch based on duplicity file listing + +**Version 0.1.3 (2010-01-13)** + +- fixed: (again) prefetch restore of files split between volumes +- fixed: missing import sys +- fixed: spurious linebreaks in debug log + +**Version 0.1.2 (2010-01-13)** + +- fixed: prefetch_recent when joblist contains a single non full job +- fixed: prefetch restore of files split between volumes +- added: copyright, trademark and license blurbs + +**Version 0.1.1 (2010-01-07)** + +- workaround: subtle extraction bug (fix forthcoming) + +**Version 0.1.1 (2010-01-06)** + +- initial public release + +Bugs ---- Please report problems via the **BaculaFS** issue tracking system: `<http://code.google.com/p/baculafs/issues/list>`_ -LICENSE +Credits +------- + +**BaculaFS** contains SQL queries that were adapted from Bacula, +Copyright |(C)| 2000-2010 Free Software Foundation Europe e.V. + +License ------- **BaculaFS** is free software: you can redistribute it and/or modify @@ -353,5 +534,7 @@ along with this program. If not, see `<http://www.gnu.org/licenses/>`_. +.. [*] Bacula is a registered trademark of Kern Sibbald. +.. |(C)| unicode:: 0xA9 .. copyright sign

@@ -1,3 +1,27 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + class Base64 : ''' Bacula specific implementation of a base64 decoder

@@ -1,3 +1,27 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + from datetime import datetime from SQL import * @@ -74,7 +98,7 @@ diff_jobs = self.db.query(SQL.temp) self.db.query(SQL.incr_jobs_temp % (diff_jobs[-1][1], self.datetime, self.client_id, self.fileset[1]), fetch=False) jobs = self.db.query(SQL.jobs) - self.most_recent_jobid = jobs[-1][0] if len(jobs) > 1 else -1 + self.most_recent_jobid = jobs[-1][0] if jobs[-1][1] != 'F' else -1 # select files from the most recent job only if select_recent_job and self.most_recent_jobid > 0 : jobs = [jobs[-1]]

@@ -1,4 +1,29 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + import os +from getpass import getpass from Base64 import * from SQL import * @@ -8,12 +33,11 @@ This class shields the rest of the code from the pesky details of actually accessing one of the supported databases. ''' - drivers = [SQL.MYSQL, SQL.POSTGRESQL, SQL.SQLITE, SQL.SQLITE3] + drivers = [SQL.MYSQL, SQL.POSTGRESQL, SQL.SQLITE3] default_database = { SQL.MYSQL: 'bacula', SQL.POSTGRESQL: 'bacula', - SQL.SQLITE: '/var/lib/bacula/bacula.db', SQL.SQLITE3: '/var/lib/bacula/bacula.db' } @@ -28,18 +52,21 @@ self.driver = driver if not database : database = Database.default_database[self.driver] + # user supplied empty password -> prompt + if password == '' : + password = getpass() + # user supplied no password -> query environment or use empty string + if password == None : + try: + password = os.environ['DATABASE_PASSWORD'] + except KeyError: + password = '' if self.driver == SQL.MYSQL : from MySQLdb import connect self.connection = connect(host=host, port=port, user=username, passwd=password, db=database) elif self.driver == SQL.POSTGRESQL : from psycopg2 import connect self.connection = connect(host=host, port=port, user=username, password=password, database=database) - elif self.driver == SQL.SQLITE : - from sqlite import connect - database = os.path.expanduser(database) - if not os.path.isfile(database) or not os.access(database, os.R_OK) : - raise RuntimeError, 'cannot read from file %s' % database - self.connection = connect(database) elif self.driver == SQL.SQLITE3 : from sqlite3 import connect database = os.path.expanduser(database) @@ -64,10 +91,10 @@ def query(self, sql, fetch = True) : ''' - Execute SQL and fetch all results + Execute SQL and fetch all results as a list ''' self.logger.debug(sql) self.cursor.execute(sql) if fetch : - return self.cursor.fetchall() + return list(self.cursor.fetchall())

@@ -1,6 +1,31 @@ -__version__ = '0.1.0' + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + +__version__ = '0.1.7' import os +import sys import stat import errno import copy @@ -10,10 +35,14 @@ import traceback import pexpect import fcntl +import time +import re +import binascii from LogFile import * from Database import * from Catalog import * +from SQL import * # pull in some spaghetti to make this stuff work without fuse-py being installed try: @@ -71,7 +100,18 @@ class FileSystem(Fuse) : - null_stat = fuse.Stat(st_mode = stat.S_IFDIR | 0755, st_nlink = 2, st_ino = -1) + null_stat = fuse.Stat(st_mode=stat.S_IFDIR | 0755, + st_ino=0, + st_dev=0, + st_nlink=2, + st_uid=0, + st_gid=0, + st_size=0, + st_atime=0, + st_mtime=0, + st_ctime=0, + st_blksize=0, + st_rdev=0) bacula_stat_fields = ['st_dev', 'st_ino', @@ -90,7 +130,7 @@ 'st_flags', 'st_streamid'] - fuse_stat_fields = dir(fuse.Stat()) + fuse_stat_fields = [attr for attr in dir(null_stat) if attr.startswith('st_')] xattr_prefix = 'user.baculafs.' xattr_fields = ['FileIndex', 'JobId', 'LStat', 'MD5'] @@ -115,14 +155,14 @@ self._initialized = False # default option values - self.logging = 'warning' + self.logging = 'info' self.syslog = False - self.driver = 'sqlite3' + self.driver = SQL.SQLITE3 self.database = None self.host = 'localhost' self.port = 0 self.username = 'bacula' - self.password = '' + self.password = None self.conf = '/etc/bacula/bacula-sd.conf' self.client = '' self.fileset = None @@ -130,6 +170,7 @@ self.datetime = None self.recent_job = False self.joblist = None + self.cache_prefix = None self.user_cache_path = None self.cleanup = False self.move_root = False @@ -138,7 +179,15 @@ self.prefetch_symlinks = False self.prefetch_recent = False self.prefetch_diff = None + self.prefetch_difflist = None + self.prefetch_list = None self.prefetch_everything = False + self.batch_mode = False + self.batch_list = False + self.batch_bsr = False + self.batch_extract = False + self.use_ino = False + self.max_ino = 0 self.dirs = { '/': { '': (FileSystem.null_stat,) } } self._bextract_status = copy.deepcopy(FileSystem.bextract_done) @@ -188,6 +237,19 @@ self.dirs[head][tail] = (FileSystem.null_stat,) self._add_parent_dirs(head) + def _update_inodes(self, head) : + ''' + generate unique st_ino for each missing st_ino + ''' + for tail in self.dirs[head] : + if self.dirs[head][tail][-1].st_ino == 0 : + if len(self.dirs[head][tail]) == 1: + self.dirs[head][tail] = (copy.deepcopy(FileSystem.null_stat),) + self.max_ino += 1 + self.dirs[head][tail][-1].st_ino = self.max_ino + subdir = '%s%s/' % (head, tail) + if subdir in self.dirs : + self._update_inodes(subdir) def _extract(self, path_list) : ''' @@ -227,9 +289,7 @@ rc, sig = self._bextract(items) # it seems that bextract does not restore mtime for symlinks # so we create a normal file with same mtime as stored symlink - # (note that we only use that file if the cache path was - # supplied by the user) - if rc == 0 : + if rc == 0 and not self.batch_mode : for item in items : if item[0] : symlinkfile = item[0][0] @@ -279,19 +339,14 @@ is_symlink = stat.S_ISLNK(bs.st_mode) found = False if os.path.exists(realpath) or os.path.lexists(realpath) : - # do not trust user supplied cache path: # make sure that stat info of realpath matches path - if self.user_cache_path : - s = os.lstat(realpath) - conds = [getattr(s, attr) == getattr(bs, attr) - for attr in ['st_mode', 'st_uid', 'st_gid', 'st_size', 'st_mtime']] - if is_symlink : - conds[-1] = (os.path.exists(symlinkpath) and - bs.st_mtime == os.stat(symlinkpath).st_mtime) - found = all(conds) - else : - found = True - if found : + s = os.lstat(realpath) + conds = [getattr(s, attr) == getattr(bs, attr) + for attr in ['st_mode', 'st_uid', 'st_gid', 'st_size', 'st_mtime']] + if is_symlink : + conds[-1] = (os.path.exists(symlinkpath) and + bs.st_mtime == os.stat(symlinkpath).st_mtime) + if all(conds) : return realpath, None, None # generate list of volumes for path fileindex, jobid = self.dirs[head][tail][0:2] @@ -300,11 +355,12 @@ volumes = [[volume[1], # 0-Volume volume[2], # 1-MediaType self.device, # 2-Device - jobs[0][1], # 3-VolSessionId - jobs[0][2], # 4-VolSessionTime - (volume[5] << 32) | volume[7], # 5-VolAddr: StartAddr - (volume[6] << 32) | volume[8], # 6-VolAddr: EndAddr - fileindex] # 7-FileIndex + jobs[0][0], # 3-JobId + jobs[0][1], # 4-VolSessionId + jobs[0][2], # 5-VolSessionTime + (volume[5] << 32) | volume[7], # 6-VolAddr: StartAddr + (volume[6] << 32) | volume[8], # 7-VolAddr: EndAddr + fileindex] # 8-FileIndex for volume in self.catalog.volumes if (volume[0] == jobid and volume[3] <= fileindex and @@ -369,8 +425,31 @@ ''' extract list of items from Bacula storage device ''' + if self.batch_list : + for item in items : + print item[1] + if (not self.batch_bsr and + not self.batch_extract) : + return (0, 0) + bsrpath = self._write_bsr(items) - cmd = 'bextract -b "%s" -c "%s" "%s" "%s"' % (bsrpath, self.conf, self.device, self.cache_path) + + if self.batch_bsr : + bsrfile = open(bsrpath, 'rt') + for line in bsrfile : + sys.stdout.write(line) + sys.stdout.flush() + bsrfile.close() + if not self.batch_extract : + return (0, 0) + + if self.batch_extract : + makedirs(self.fuse_args.mountpoint) + + cmd = 'bextract -b "%s" -c "%s" "%s" "%s"' % (bsrpath, + self.conf, + self.device, + self.cache_path if not self.batch_extract else self.fuse_args.mountpoint) self.logger.debug(cmd) self._bextract_set_status({'path': items[0][1], @@ -393,6 +472,7 @@ i = child.expect([self.fail_pattern, pexpect.EOF], timeout=None, searchwindowsize=200) + self.logfile.flush(flush_tail = True) if i == 0 : # count retries if missing == child.match.groups()[0] : @@ -434,21 +514,69 @@ self._bextract_increment_counter('failures', 1) return (child.exitstatus, child.signalstatus) + def _group_by_volume(self, items) : + ''' + return items grouped by volume + ''' + # group volumes + volumes = [] + for item in items : + for v in item[-1] : + found = False + findex = v[-1] + for vindex in xrange(0,len(volumes)) : + volume = volumes[vindex] + if not any(map(cmp, v[:-1], volume[:-1])) : + volume[-1].append(findex) + found = True + break + if not found : + volumes.append(v[:-1] + [[v[-1]]]) + + # compact list of file indices + for volume in volumes : + volume[-1] = list(set(volume[-1])) + volume[-1].sort() + l = len(volume[-1]) + findex = volume[-1][0] + findices = [(findex, findex)] + for idx in volume[-1][1:] : + next_idx = findices[-1][-1] + 1 + if idx == next_idx : + findices[-1] = (findices[-1][0], idx) + else : + findices.append((idx,idx)) + volume[-1] = findices + volume.append(l) + + # reorder volumes to ensure correct handling of + # files spanning multiple volumes + volumes.sort(cmp = lambda a,b : \ + (cmp(a[3],b[3]) or + cmp(a[8][-1][-1], b[8][0][0]))) + + return volumes + def _write_bsr(self, items) : ''' generate bsr for items to be extracted ''' bsrfd, bsrpath = tempfile.mkstemp(suffix='.bsr', dir=self.cache_bsrpath, text=True) - for item in items : - for volume in item[-1] : - os.write(bsrfd, 'Volume="%s"\n' % volume[0]) - os.write(bsrfd, 'MediaType="%s"\n' % volume[1]) - os.write(bsrfd, 'Device="%s"\n' % volume[2]) - os.write(bsrfd, 'VolSessionId=%d\n' % volume[3]) - os.write(bsrfd, 'VolSessionTime=%d\n' % volume[4]) - os.write(bsrfd, 'VolAddr=%d-%d\n' % (volume[5],volume[6])) - os.write(bsrfd, 'FileIndex=%d\n' % volume[7]) - os.write(bsrfd, 'Count=1\n') + volumes = self._group_by_volume(items) + for volume in volumes : + os.write(bsrfd, 'Volume="%s"\n' % volume[0]) + os.write(bsrfd, 'MediaType="%s"\n' % volume[1]) + os.write(bsrfd, 'Device="%s"\n' % volume[2]) + os.write(bsrfd, 'VolSessionId=%d\n' % volume[4]) + os.write(bsrfd, 'VolSessionTime=%d\n' % volume[5]) + if not self.bsr_compat : + os.write(bsrfd, 'VolAddr=%d-%d\n' % (volume[6],volume[7])) + for findex in volume[8] : + if findex[0] == findex[1] : + os.write(bsrfd, 'FileIndex=%d\n' % findex[0]) + else : + os.write(bsrfd, 'FileIndex=%d-%d\n' % findex) + os.write(bsrfd, 'Count=%d\n' % volume[9]) os.close(bsrfd) return bsrpath @@ -489,14 +617,22 @@ self.logger.warning(traceback.format_exc()) self.logfile = LogFile(self.logger, logging.DEBUG) - - def initialize(self): + def initialize(self, version) : ''' initialize database, catalog ''' self._setup_logging() + # batch mode + self.batch_mode = (self.batch_list or + self.batch_bsr or + self.batch_extract) + # disable INFO level logging in batch mode + if self.batch_mode and self.loglevel == logging.INFO : + self.loglevel = logging.WARNING + self.logger.setLevel(self.loglevel) + self.logger.info('Populating file system ... ') # setup cache @@ -511,6 +647,10 @@ self.cache_symlinks = os.path.normpath(self.cache_prefix + '/symlinks') makedirs(self.cache_symlinks) + # test for old version (2.x) of bacula + self.bsr_compat = int(version[0]) < 3 + if self.bsr_compat : + self.logger.debug('Detected old Bacula: %s' % version) # test access to sd conf file open(self.conf, 'r').close() # init bextract failure pattren @@ -534,12 +674,15 @@ self.db.close() prefetches = [] + difflist = {} # validate prefetch conditions if self.prefetch_everything : self.prefetch_recent = False self.prefetch_regex = None self.prefetch_diff = None + self.prefetch_difflist = None + self.prefetch_list = None self.prefetch_symlinks = True if self.prefetch_regex : try : @@ -560,10 +703,37 @@ # can't access target directory: show traceback and ignore self.logger.warning(traceback.format_exc()) self.prefetch_diff = None + if self.prefetch_difflist : + self.prefetch_difflist = os.path.normpath(os.path.expanduser(self.prefetch_difflist)) + try : + difflistfile = sys.stdin if self.prefetch_difflist == '-' else open(self.prefetch_difflist, 'rt') + for line in difflistfile.readlines(): + date = ' '.join(line.split()[:5]) + difflist[line[(len(date) + 1):].strip()] = time.strptime(date, '%a %b %d %H:%M:%S %Y') + difflistfile.close() + self.prefetch_symlinks = True + except : + # can't access/parse difflist: show traceback and ignore + self.logger.warning(traceback.format_exc()) + self.prefetch_difflist = None + if self.prefetch_list : + self.prefetch_list = os.path.normpath(os.path.expanduser(self.prefetch_list)) + try : + listfile = sys.stdin if self.prefetch_list == '-' else open(self.prefetch_list, 'rt') + matchlist = [line.strip() for line in listfile.readlines()] + listfile.close() + self.prefetch_symlinks = True + except : + # can't access/parse list: show traceback and ignore + self.logger.warning(traceback.format_exc()) + self.prefetch_list = None if self.prefetch_recent : self.prefetch_symlinks = True if self.prefetch_symlinks : self.prefetch_attrs = True + if 'use_ino' in self.fuse_args.optlist: + self.use_ino = True + self.prefetch_attrs = True # must figure out max st_ino for file in files : head = file[0] @@ -574,6 +744,10 @@ # make file entry if self.prefetch_attrs : entry = file[2:] + self._bacula_stat(file[-2]) + # find max st_ino + if self.use_ino: + if entry[-1].st_ino > self.max_ino : + self.max_ino = entry[-1].st_ino # detemine if we need to prefetch this entry filepath = head + tail if (not stat.S_ISDIR(entry[-1].st_mode) and @@ -581,9 +755,14 @@ (self.prefetch_recent and file[3] == self.catalog.most_recent_jobid) or (self.prefetch_regex and - regex.match(filepath)) or + regex.search(filepath)) or (self.prefetch_diff and - not self._match_stat(self.prefetch_diff + filepath, entry[-1])) or + not self._match_stat(self.prefetch_diff + filepath, entry[-1])) or + (self.prefetch_difflist and + (filepath[1:] not in difflist or + difflist[filepath[1:]][:-1] != time.localtime(entry[-1].st_mtime)[:-1])) or + (self.prefetch_list and + filepath in matchlist) or (self.prefetch_symlinks and stat.S_ISLNK(entry[-1].st_mode)))) : prefetches.append(filepath) @@ -600,13 +779,17 @@ # and finally self.dirs[head][tail] = entry + # fix st_ino + if self.use_ino: + self._update_inodes('/') + npf = len(prefetches) if npf > 0 : self.logger.info('Prefetching %d objects ... ' % npf) self._extract(prefetches) - self.logger.info('Cache directory is: %s' % self.cache_prefix) + self.logger.debug('Cache directory is: %s' % self.cache_prefix) self.joblist = ' '.join([str(job[0]) for job in self.catalog.jobs]) - self.logger.info('Jobs in file system: %s' % self.joblist) + self.logger.debug('Job ids in file system: %s' % self.joblist) self.logger.info('BaculaFS ready (%d files).' % len(files)) self._initialized = True @@ -615,7 +798,7 @@ ''' remove cache directory if required ''' - if self.cleanup and not self.user_cache_path : + if self.cleanup and not self.user_cache_path and self.cache_prefix : self.logger.info('removing cache directory: %s' % self.cache_prefix) shutil.rmtree(self.cache_prefix, ignore_errors = True) @@ -653,6 +836,9 @@ len(self.dirs[head][tail]) != 1 and n in FileSystem.xattr_fields) : val = str(self.dirs[head][tail][FileSystem.xattr_fields.index(n)]) + if n == 'MD5' and val != '0': + l = len(val) + val = binascii.b2a_hex(binascii.a2b_base64(val+'='*((l*3+8)/3-l)+'\n')) # padding # attribute not found if val == None : return -errno.ENODATA @@ -719,8 +905,11 @@ yield fuse.Direntry(key) for key in self.dirs[path].keys() : if len(key) > 0: - bs = self.getattr(path + key) - ino = bs.st_ino if bs.st_ino != 0 else -1 + if self.use_ino: + bs = self.getattr(path + key) + ino = bs.st_ino + else : + ino = 0 yield fuse.Direntry(key, ino=ino) def readlink(self, path): @@ -797,7 +986,7 @@ server.parser.add_option(mountopt="username", metavar="USERNAME", default=server.username, help="database user name [default: %default]") server.parser.add_option(mountopt="password", metavar="PASSWORD", default=server.password, - help="database password") + help="database password (use '-o password= ' to get a password prompt; if not provided, the password is read from the DATABASE_PASSWORD environment variable)") server.parser.add_option(mountopt="conf", metavar="PATH", default=server.conf, help="storage daemon configuration file [default: %default]") server.parser.add_option(mountopt="client", metavar="CLIENT", default=server.client, @@ -826,8 +1015,18 @@ help="extract contents of most recent non-full job upon filesystem initialization (implies prefetch_symlinks) [default: %default]") server.parser.add_option(mountopt="prefetch_diff", metavar="PATH", default=server.prefetch_diff, help="extract files that do not match files at PATH (hint: speeds up rsync; implies prefetch_symlinks)") + server.parser.add_option(mountopt="prefetch_difflist", metavar="DIFFLIST", default=server.prefetch_difflist, + help="extract files that do not match files in DIFFLIST (list line format: 'Day Mon DD hh:mm:ss YYYY PATH'; use '-' to read from standard input; hint: format matches output of 'duplicity list-current-files -v0 target_url'; implies prefetch_symlinks)") + server.parser.add_option(mountopt="prefetch_list", metavar="LIST", default=server.prefetch_list, + help="extract files that match files in LIST (list should contains one absolute file path per line; use '-' to read from standard input; implies prefetch_symlinks)") server.parser.add_option(mountopt="prefetch_everything", action="store_true", default=server.prefetch_everything, help="extract everything upon filesystem initialization (complete restore to cache) [default: %default]") + server.parser.add_option(mountopt="batch_list", action="store_true", default=server.batch_list, + help="list files to be prefetched and exit [default: %default]") + server.parser.add_option(mountopt="batch_bsr", action="store_true", default=server.batch_bsr, + help="dump contnets of bsr file for extracting prefetched files and exit [default: %default]") + server.parser.add_option(mountopt="batch_extract", action="store_true", default=server.batch_extract, + help="extract prefetched files to mount point and exit [default: %default]") server.parser.add_option(mountopt="user_cache_path", metavar="PATH", default=server.user_cache_path, help="user specified cache path (hint: combine this with one of the prefetch options) [default: %default]") server.parser.add_option(mountopt="logging", choices=LOGGING_LEVELS.keys(), metavar='|'.join(LOGGING_LEVELS.keys()), default=server.logging, @@ -844,13 +1043,13 @@ # we initialize before main (i.e. not in fsinit) so that # any failure here aborts the mount try : - server.initialize() + server.initialize(bacula_version) except : - traceback.print_exc() server.shutdown() raise - server.main() + if not server.batch_mode : + server.main() # we shutdown after main, i.e. not in fsshutdown, because # calling fsshutdown with multithreaded==True seems to cause

@@ -1,3 +1,27 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + import logging import logging.handlers @@ -14,12 +38,25 @@ def __init__(self, logger, level) : self.logger = logger self.level = level + self.tail = '' def write(self, message) : - for line in message.splitlines() : + lines = (self.tail + message).splitlines() + for line in lines[:-1] : self.logger.log(self.level, line) + if message.endswith('\n') : + self.logger.log(self.level, lines[-1]) + self.tail = '' + elif len(lines) > 0 : + self.tail = lines[-1] + else : + self.tail = '' + - def flush(self) : + def flush(self, flush_tail = False) : + if flush_tail and self.tail : + self.logger.log(self.level, self.tail) + self.tail = '' for handler in self.logger.handlers : handler.flush()

@@ -1,3 +1,27 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + class SQL : ''' Holds all SQL statements used by baculafs. @@ -6,7 +30,6 @@ MYSQL = 'mysql' POSTGRESQL = 'postgresql' - SQLITE = 'sqlite' SQLITE3 = 'sqlite3' clients = 'SELECT Client.Name,ClientId FROM Client' @@ -54,21 +77,6 @@ VolSessionTime INTEGER) ''', - SQLITE: ''' - CREATE TEMPORARY TABLE temp ( - JobId INTEGER UNSIGNED NOT NULL, - JobTDate BIGINT UNSIGNED, - ClientId INTEGER UNSIGNED, - Level CHAR, - JobFiles INTEGER UNSIGNED, - JobBytes BIGINT UNSIGNED, - StartTime TEXT, - VolumeName TEXT, - StartFile INTEGER UNSIGNED, - VolSessionId INTEGER UNSIGNED, - VolSessionTime INTEGER UNSIGNED) - ''', - SQLITE3: ''' CREATE TEMPORARY TABLE temp ( JobId INTEGER UNSIGNED NOT NULL, @@ -97,12 +105,6 @@ JobTDate BIGINT) ''', - SQLITE: ''' - CREATE TEMPORARY TABLE temp1 ( - JobId INTEGER UNSIGNED NOT NULL, - JobTDate BIGINT UNSIGNED) - ''', - SQLITE3: ''' CREATE TEMPORARY TABLE temp1 ( JobId INTEGER UNSIGNED NOT NULL, @@ -191,7 +193,7 @@ AND FileSet.FileSet='%s' ''' - jobs = 'SELECT DISTINCT JobId,StartTime FROM temp ORDER BY StartTime ASC' + jobs = 'SELECT DISTINCT JobId,Level,StartTime FROM temp ORDER BY StartTime ASC' base_jobs = ''' SELECT DISTINCT BaseJobId @@ -257,32 +259,6 @@ -- dummy comment for chomping extra parameter: %s ''', - SQLITE: ''' - SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId, - File.FilenameId AS FilenameId, LStat, MD5 - FROM Job, File, ( - SELECT MAX(JobTDate) AS JobTDate, PathId, FilenameId - FROM ( - SELECT JobTDate, PathId, FilenameId - FROM File JOIN Job USING (JobId) - WHERE File.JobId IN (%s) - UNION ALL - SELECT JobTDate, PathId, FilenameId - FROM BaseFiles - JOIN File USING (FileId) - JOIN Job ON (BaseJobId = Job.JobId) - WHERE BaseFiles.JobId IN (%s) - ) AS tmp GROUP BY PathId, FilenameId - ) AS T1 - WHERE (Job.JobId IN ( - SELECT DISTINCT BaseJobId FROM BaseFiles WHERE JobId IN (%s)) - OR Job.JobId IN (%s)) - AND T1.JobTDate = Job.JobTDate - AND Job.JobId = File.JobId - AND T1.PathId = File.PathId - AND T1.FilenameId = File.FilenameId - ''', - SQLITE3: ''' SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId, File.FilenameId AS FilenameId, LStat, MD5 @@ -313,13 +289,13 @@ SELECT JobId,VolSessionId,VolSessionTime, PoolId,StartTime,EndTime,JobFiles,JobBytes,JobTDate,Job,JobStatus, Type,Level,ClientId,Name,PriorJobId,RealEndTime,FileSetId, - SchedTime,RealEndTime,ReadBytes,HasBase + SchedTime,RealEndTime,HasBase FROM Job WHERE JobId IN (%s) ''' volumes = ''' SELECT JobMedia.JobId,VolumeName,MediaType,FirstIndex,LastIndex,StartFile, - JobMedia.EndFile,StartBlock,JobMedia.EndBlock,Copy, + JobMedia.EndFile,StartBlock,JobMedia.EndBlock, Slot,StorageId,InChanger FROM JobMedia,Media WHERE JobMedia.JobId IN (%s) AND JobMedia.MediaId=Media.MediaId ORDER BY JobMedia.JobId,VolIndex,JobMediaId

@@ -1,7 +1,31 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + """Expose the Bacula catalog as a FUSE file system. """ from FileSystem import __version__, main -__all__ = ["FileSystem", "Database", "Catalog", "Base64", "SQL"] +__all__ = ["FileSystem", "Database", "Catalog", "Base64", "SQL", "LogFile"]

@@ -1,5 +1,28 @@ #! /usr/bin/env python +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + '''BaculaFS mount script '''

@@ -0,0 +1,6 @@ +import commands +baculafs_mount_points = [tokens[2] for tokens in + [line.split() for line in commands.getoutput('mount -v').split('\n')] + if tokens[4].endswith('baculafs')] +if self.fuse_args.mountpoint in baculafs_mount_points : + raise RuntimeError, 'mountpoint already in use by BaculaFS'

@@ -1,5 +1,28 @@ #!/usr/bin/python +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + from setuptools import setup, find_packages from baculafs import __version__ @@ -23,6 +46,7 @@ install_requires=['fuse-python>=0.2','pexpect>=2.3','MySQL-python>=1.2.2','psycopg2>=2.0.13'], classifiers = [ "Development Status :: 3 - Alpha", + "Topic :: System :: Filesystems", "Topic :: System :: Archiving :: Backup", "Intended Audience :: System Administrators", "Environment :: No Input/Output (Daemon)",

[-] [+]	Changed	baculafs.changes
@@ -1,4 +1,9 @@ ------------------------------------------------------------------- +Mon Oct 3 09:36:36 UTC 2011 - cs@linux-administrator.com + +- update to version 0.1.7 + +------------------------------------------------------------------- Mon May 31 17:45:58 UTC 2010 - cs@linux-administrator.com - update to version 0.1.4
[-] [+]	Changed	baculafs.spec ^
@@ -5,7 +5,7 @@ # norootforbuild Name: baculafs -Version: 0.1.4 +Version: 0.1.7 Release: 1 Url: http://pypi.python.org/packages/source/B/BaculaFS/BaculaFS-%{version}.tar.gz License: GPL v2 or later
	Deleted	BaculaFS-0.1.4.tar.gz ^
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/BaculaFS.egg-info/PKG-INFO ^
@@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: BaculaFS -Version: 0.1.0 +Version: 0.1.7 Summary: Bacula Filesystem in USErspace Home-page: http://code.google.com/p/baculafs Author: Avi Rozen @@ -10,64 +10,67 @@ BaculaFS ======== - BaculaFS_ - Exposes the Bacula_ catalog and storage as a Filesystem in + BaculaFS_ - Exposes the Bacula_ []_ catalog and storage as a Filesystem in USErspace (FUSE_). .. _BaculaFS: http://code.google.com/p/baculafs .. _Bacula: http://www.bacula.org .. _FUSE: http://fuse.sourceforge.net/ - Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> + Copyright \|(C)\| 2009, 2010 Avi Rozen <avi.rozen@gmail.com> - INTRODUCTION + .. contents:: + + Introduction ------------ - BaculaFS* is a tool, independent of Bacula, that represents the - Bacula catalog and backup storage media as a read-only filesystem in - userspace. + BaculaFS is a tool, developed independently of Bacula, that + represents the Bacula catalog and backup storage media as a read-only + filesystem in userspace. BaculaFS is specifically designed to cater for the following use-cases: - maintaining a remote snapshot of the files in the backup storage - using `rsync`_ + using `rsync`_ or `duplicity`_ - auditing the contents of backup jobs, without resorting to SQL - queries + queries - comparing backup jobs (using several mount points) - Note that BaculaFS is a maintenance tool - it's operation may + Note that BaculaFS is a maintenance tool - its operation may interfere with the normal operation of a live Bacula setup (see the - LIMITATIONS_ section below). + Limitations_ section below). .. _rsync: http://samba.anu.edu.au/rsync/ + .. _duplicity: http://www.nongnu.org/duplicity/ - REQUIREMENTS + Requirements ------------ BaculaFS has been tested with the following set of required software packages: - + Bacula 3.0.2 with one of the following database backends: - - * SQLite_ 3.6.21 - * MySQL_ 5.4.1 - * PostgreSQL_ 8.3.9 + + Bacula 5.0.2 with one of the following database backends: + * SQLite_ 3.7.3 + * MySQL_ 5.1.49 + * PostgreSQL_ 8.4.5 + + FUSE_ components: - * Python FUSE 0.2 - * FUSE library 2.8.1 - * fusermount 2.8.1 - * FUSE kernel interface 7.12 - - + Python_ 2.5.4, with the following additional libraries: - - * MySQLdb_ 1.2.2 - * psycopg2_ 2.0.13 - * pexpect_ 2.3 - + * Python FUSE 0.2.1 + * FUSE library 2.8.4 + * fusermount 2.8.4 + * FUSE kernel interface 7.12 + + + Python_ 2.6.6, with the following additional libraries: + + * MySQLdb_ 1.2.2 + * psycopg2_ 2.2.1 + * pexpect_ 2.3 + + attr_ extended attributes utilities 2.4.44 BaculaFS requires the following in order to function: @@ -75,7 +78,7 @@ + access to the database that's used to store the Bacula catalog + access to the Bacula storage device + access to the Bacula ``bextract`` utility (bundled with the Bacula - storage daemon installation) + storage daemon installation) .. _SQLite: http://www.sqlite.org/ .. _MySQL: http://www.mysql.com/ @@ -88,7 +91,7 @@ .. _attr: http://savannah.nongnu.org/projects/attr - INSTALLATION + Installation ------------ Extract the source code archive to a temporary directory, ``cd`` to @@ -96,117 +99,136 @@ :: - python setup.py install + python setup.py install - USAGE + Usage ----- :: - baculafs [mountpoint] [options] - - Options: - --version show program's version number and exit - -h, --help show this help message and exit - -o opt,[opt...] mount options - -o driver=mysql\|postgresql\|sqlite\|sqlite3 - database driver [default: sqlite3] - -o host=HOST database server address [default: localhost] - -o port=PORT database server port - -o database=PATH database name [default: bacula] - -o username=USERNAME database user name [default: bacula] - -o password=PASSWORD database password - -o conf=PATH storage daemon configuration file [default: - /etc/bacula/bacula-sd.conf] - -o client=CLIENT file daemon name - -o fileset=FILESET backup fileset - -o device=DEVICE storage device name [default: FileStorage] - -o datetime='YYYY-MM-DD hh:mm:ss' - snapshot date/time [default: now] - -o recent_job select contents of most recent job only [default: - False] - -o joblist='JOBID1 JOBID2 ...' - select contents of specified list of jobs - -o cleanup clean cache directory upon umount [default: False] - -o move_root make absolute path symlinks point to path under - mount point [default: False] - -o prefetch_attrs read and parse attributes for all files upon - filesystem initialization [default: False] - -o prefetch_symlinks extract all symbolic links upon filesystem - initialization (implies prefetch_attrs) [default: - False] - -o prefetch_regex=REGEX - extract all objects that match REGEX upon - filesystem initialization (implies prefetch_attrs) - -o prefetch_recent extract contents of most recent non-full job upon - filesystem initialization (implies - prefetch_symlinks) [default: False] - -o prefetch_diff=PATH extract files that do not match files at PATH - (hint: speeds up rsync; implies prefetch_symlinks) - -o prefetch_everything - extract everything upon filesystem initialization - (complete restore to cache) [default: False] - -o user_cache_path=PATH - user specified cache path (hint: combine this with - one of the prefetch options) [default: none] - -o logging=debug\|info\|warning\|critical\|error - logging level [default: warning] - -o syslog log to both syslog and console [default: False] - - FUSE options: - -d -o debug enable debug output (implies -f) - -f foreground operation - -s disable multi-threaded operation - - -o allow_other allow access to other users - -o allow_root allow access to root - -o nonempty allow mounts over non-empty file/dir - -o default_permissions enable permission checking by kernel - -o fsname=NAME set filesystem name - -o subtype=NAME set filesystem type - -o large_read issue large read requests (2.4 only) - -o max_read=N set maximum size of read requests - - -o hard_remove immediate removal (don't hide files) - -o use_ino let filesystem set inode numbers - -o readdir_ino try to fill in d_ino in readdir - -o direct_io use direct I/O - -o kernel_cache cache files in kernel - -o [no]auto_cache enable caching based on modification times (off) - -o umask=M set file permissions (octal) - -o uid=N set file owner - -o gid=N set file group - -o entry_timeout=T cache timeout for names (1.0s) - -o negative_timeout=T cache timeout for deleted names (0.0s) - -o attr_timeout=T cache timeout for attributes (1.0s) - -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout) - -o intr allow requests to be interrupted - -o intr_signal=NUM signal to send on interrupt (10) - -o modules=M1[:M2...] names of modules to push onto filesystem stack - - -o max_write=N set maximum size of write requests - -o max_readahead=N set maximum readahead - -o async_read perform reads asynchronously (default) - -o sync_read perform reads synchronously - -o atomic_o_trunc enable atomic open+truncate support - -o big_writes enable larger than 4kB writes - -o no_remote_lock disable remote file locking - - Module options: - - [subdir] - -o subdir=DIR prepend this directory to all paths (mandatory) - -o [no]rellinks transform absolute symlinks to relative - - [iconv] - -o from_code=CHARSET original encoding of file names (default: UTF-8) - -o to_code=CHARSET new encoding of the file names (default: UTF-8) - + baculafs [mountpoint] [options] + + Options: + --version show program's version number and exit + -h, --help show this help message and exit + -o opt,[opt...] mount options + -o driver=mysql\|postgresql\|sqlite3 + database driver [default: sqlite3] + -o host=HOST database server address [default: localhost] + -o port=PORT database server port + -o database=PATH database name [default: bacula] + -o username=USERNAME database user name [default: bacula] + -o password=PASSWORD database password (use '-o password= ' to get a + password prompt; if not provided, the password is + read from the DATABASE_PASSWORD environment + variable) + -o conf=PATH storage daemon configuration file [default: + /etc/bacula/bacula-sd.conf] + -o client=CLIENT file daemon name + -o fileset=FILESET backup fileset + -o device=DEVICE storage device name [default: FileStorage] + -o datetime='YYYY-MM-DD hh:mm:ss' + snapshot date/time [default: now] + -o recent_job select contents of most recent job only [default: + False] + -o joblist='JOBID1 JOBID2 ...' + select contents of specified list of jobs + -o cleanup clean cache directory upon umount [default: False] + -o move_root make absolute path symlinks point to path under + mount point [default: False] + -o prefetch_attrs read and parse attributes for all files upon + filesystem initialization [default: False] + -o prefetch_symlinks extract all symbolic links upon filesystem + initialization (implies prefetch_attrs) [default: + False] + -o prefetch_regex=REGEX + extract all objects that match REGEX upon + filesystem initialization (implies prefetch_attrs) + -o prefetch_recent extract contents of most recent non-full job upon + filesystem initialization (implies + prefetch_symlinks) [default: False] + -o prefetch_diff=PATH extract files that do not match files at PATH + (hint: speeds up rsync; implies prefetch_symlinks) + -o prefetch_difflist=DIFFLIST + extract files that do not match files in DIFFLIST + (list line format: 'Day Mon DD hh:mm:ss YYYY PATH'; + use '-' to read from standard input; hint: format + matches output of 'duplicity list-current-files -v0 + target_url'; implies prefetch_symlinks) + -o prefetch_list=LIST extract files that match files in LIST (list should + contains one absolute file path per line; use '-' + to read from standard input; implies + prefetch_symlinks) + -o prefetch_everything + extract everything upon filesystem initialization + (complete restore to cache) [default: False] + -o batch_list list files to be prefetched and exit [default: + False] + -o batch_bsr dump contnets of bsr file for extracting prefetched + files and exit [default: False] + -o batch_extract extract prefetched files to mount point and exit + [default: False] + -o user_cache_path=PATH + user specified cache path (hint: combine this with + one of the prefetch options) [default: none] + -o logging=debug\|info\|warning\|critical\|error + logging level [default: info] + -o syslog log to both syslog and console [default: False] + + FUSE options: + -d -o debug enable debug output (implies -f) + -f foreground operation + -s disable multi-threaded operation + + -o allow_other allow access to other users + -o allow_root allow access to root + -o nonempty allow mounts over non-empty file/dir + -o default_permissions enable permission checking by kernel + -o fsname=NAME set filesystem name + -o subtype=NAME set filesystem type + -o large_read issue large read requests (2.4 only) + -o max_read=N set maximum size of read requests + + -o hard_remove immediate removal (don't hide files) + -o use_ino let filesystem set inode numbers + -o readdir_ino try to fill in d_ino in readdir + -o direct_io use direct I/O + -o kernel_cache cache files in kernel + -o [no]auto_cache enable caching based on modification times (off) + -o umask=M set file permissions (octal) + -o uid=N set file owner + -o gid=N set file group + -o entry_timeout=T cache timeout for names (1.0s) + -o negative_timeout=T cache timeout for deleted names (0.0s) + -o attr_timeout=T cache timeout for attributes (1.0s) + -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout) + -o intr allow requests to be interrupted + -o intr_signal=NUM signal to send on interrupt (10) + -o modules=M1[:M2...] names of modules to push onto filesystem stack + + -o max_write=N set maximum size of write requests + -o max_readahead=N set maximum readahead + -o async_read perform reads asynchronously (default) + -o sync_read perform reads synchronously + -o atomic_o_trunc enable atomic open+truncate support + -o big_writes enable larger than 4kB writes + -o no_remote_lock disable remote file locking + + Module options: + + [subdir] + -o subdir=DIR prepend this directory to all paths (mandatory) + -o [no]rellinks transform absolute symlinks to relative + + [iconv] + -o from_code=CHARSET original encoding of file names (default: UTF-8) + -o to_code=CHARSET new encoding of the file names (default: UTF-8) + - OPERATION + Operation --------- - INITIALIZATION + Initialization ~~~~~~~~~~~~~~ BaculaFS starts by running several SQL queries against the Bacula @@ -223,7 +245,7 @@ At this point the filesystem is ready. - CACHE + Cache ~~~~~ Opening a file for reading causes BaculaFS to run ``bextract`` in @@ -239,9 +261,9 @@ For example: - use ``-o prefetch_attrs`` for storage space usage analysis - (e.g. with Baobab_) + (e.g. with Baobab_) - use ``-o prefetch_symlinks`` for any manual filesystem traversal - with command line or GUI tools (``find``, ``mc``, etc.) + with command line or GUI tools (``find``, ``mc``, etc.) - use ``-o prefetch_diff`` with ``rsync`` .. _Baobab: http://www.marzocca.net/linux/baobab/ @@ -250,7 +272,24 @@ filesystem, with ``-o cleanup``. It may also be reused between mount operations with ``-o user_cache_path``. - EXTENDED ATTRIBUTES + Batch Mode + ~~~~~~~~~~ + + BaculaFS may be used in batch mode with ``-o batch_extract`` in + order to extract files from the Bacula storage device and then exit + without mounting the filesystem. The mountpoint specified at the + command line is then treated as the destination directory for + extracted files. + + The list of files, to be extracted in batch mode, is determined by the + various cache prefetch options, and may be dumped with ``-o + batch_list``. + + The bootstrap file that is generated, in order to extract the files, + can also be dumped to standard output with ``-o batch_bsr``. + + + Extended Attributes ~~~~~~~~~~~~~~~~~~~ BaculaFS uses extended file attributes to expose Bacula specific @@ -259,35 +298,39 @@ :: - user.baculafs.FileIndex - user.baculafs.JobId - user.baculafs.LStat - user.baculafs.MD5 + user.baculafs.FileIndex + user.baculafs.JobId + user.baculafs.LStat + user.baculafs.MD5 + + Note that ``user.baculafs.MD5`` shows whatever digest Bacula was + configured to calculate for the file, be it MD5, SHA1, SHA256 or + SHA512. The root directory has several more attributes, that expose filesystem instance-specific information: :: - user.baculafs.cache_prefix - user.baculafs.client - user.baculafs.datetime - user.baculafs.fileset - user.baculafs.joblist + user.baculafs.cache_prefix + user.baculafs.client + user.baculafs.datetime + user.baculafs.fileset + user.baculafs.joblist and several more attributes for monitoring the file extraction process: - + :: - user.baculafs.bextract.failures - user.baculafs.bextract.path - user.baculafs.bextract.pending - user.baculafs.bextract.retries - user.baculafs.bextract.state - user.baculafs.bextract.volume + user.baculafs.bextract.failures + user.baculafs.bextract.path + user.baculafs.bextract.pending + user.baculafs.bextract.retries + user.baculafs.bextract.state + user.baculafs.bextract.volume - MISSING VOLUMES + Missing Volumes ~~~~~~~~~~~~~~~ If the storage device is a tape drive then it's possible that @@ -301,16 +344,98 @@ :: - attr -s baculafs.bextract.state -V run <mount-point> + attr -s baculafs.bextract.state -V run <mount-point> Please note that this feature has undergone only rudimentary testing. Expect breakage. - LIMITATIONS + Examples + ~~~~~~~~ + + Mount the most recent backup snapshot for Bacula client ``client-fd`` + and fileset ``client-fileset``; SQLite database backend (default): + + :: + + baculafs -o client=client-fd,fileset=client-fileset /path/to/mount/point + + Mount the contents of the specified list of backup jobs; MySQL database + backend; prompt for password and prefetch to cache all symbolic links: + + :: + + baculafs -o joblist='1001 1003',client=client-fd,fileset=client-fileset \ + -o driver=mysql,password=,prefetch_symlinks /path/to/mount/point + + Mount the contents of the most recent backup job only: + + :: + + baculafs -o recent_job,client=client-fd,fileset=client-fileset /path/to/mount/point + + Mount the contents of the fileset snapshot before the specified + date/time: + + :: + + baculafs -o datetime='2009-05-23 00:00:00' \ + -o client=client-fd,fileset=client-fileset /path/to/mount/point + + Allow other users to access filesystem, set logging level to ``debug`` + and stay in foreground, so that ``bextract`` messages may be examined; + assume a single Bacula fileset is defined for client ``client-fd``: + + :: + + baculafs -f -o allow_other,client=client-fd,logging=debug /path/to/mount/point + + Prefetch to cache files that do not exist, or whose modification time + or size differ from those at the specified directory (i.e. files that + would be sent to that directory using ``rsync``); and cleanup cache after + un-mounting: + + :: + + baculafs -o prefetch_diff=/path/to/rsync/target,cleanup \ + -o client=client-fd /path/to/mount/point + + Batch update a snapshot of the current files in backup (the filesystem + is not mounted): + + :: + + baculafs -o batch_extract,prefetch_diff=/path/to/snapshot,cleanup \ + -o client=client-fd /path/to/snapshot/ + + (this is not as accurate as mounting the filesystem, like in the + previous example, and then updating the snapshot with ``rsync`` - but + it is faster). + + + Limitations ----------- - LOCKING + Common Problems + ~~~~~~~~~~~~~~~ + + BaculaFS will abort if no backup job was run for the + client/fileset combination selected by the user. + + BaculaFS will abort if the target mount point is in use. Note, + however, that mount point availability is checked by FUSE only after + the (potentially lengthy) initialization of BaculaFS. + + Browsing a BaculaFS mount point with a file manager like Nautilus_ + is liable to be very slow. This is because the file manager reads data + from each file being listed, in order to determine its type, generate + a thumbnail etc. This, in turn, triggers BaculaFS to extract the + files, one at a time, from the Bacula storage device to its cache + directory. + + .. _Nautilus: http://live.gnome.org/Nautilus + + Locking ~~~~~~~ Access to the storage device by different instances of BaculaFS is @@ -323,28 +448,84 @@ should note: - the lock is advisory, meaning that it does not prevent the Bacula - storage daemon itself from accessing the storage device while in use - by BaculaFS + storage daemon itself from accessing the storage device while in use + by BaculaFS - depending on your setup, the lock may not work if the storage daemon - configuration file is accessed via NFS + configuration file is accessed via NFS + + + Unsupported Features + ~~~~~~~~~~~~~~~~~~~~ + + BaculaFS depends on ``bextract`` and thus inherits its + limitations: + + - BaculaFS can be used with Windows filesets, but it does not + reproduce any Windows specific file attributes + - Encrypted backup files are not supported + - BaculaFS must be started with enough permissions (typically as + ``root``) in order to allow ``bextract`` to extract files from the + Bacula storage + Changelog + --------- + Version 0.1.7 (2010-12-30) + + - fixed: compatibility issues with bacula v2.4.4 and FUSE 7.8 + - fixed: synthesize missing inode numbers with -o use_ino + - modified: decode value of user.baculafs.MD5 extended file attribute + - added: cache prefetch specified list of files + - added: changelog to README + - added: batch extract mode + + Version 0.1.6 (2010-09-19) + + - fixed cache prefetch by regex + + Version 0.1.5 (2010-07-06) + + - fixed: removed reference to obsolete db field Copy + - fixed: recent_job option with MySQL + - added: read database password from environment variable DATABASE_PASSWORD + - added: usage examples to README + + Version 0.1.4 (2010-02-07) + + - added: cache prefetch based on duplicity file listing + Version 0.1.3 (2010-01-13) - WINDOWS FILESETS - ~~~~~~~~~~~~~~~~ + - fixed: (again) prefetch restore of files split between volumes + - fixed: missing import sys + - fixed: spurious linebreaks in debug log - BaculaFS can be used with Windows backup filesets, but it does not - reproduce any Windows specific file attributes. This is because - ``bextract`` does not extract Windows specific file attributes on - Linux. + Version 0.1.2 (2010-01-13) - BUGS + - fixed: prefetch_recent when joblist contains a single non full job + - fixed: prefetch restore of files split between volumes + - added: copyright, trademark and license blurbs + + Version 0.1.1 (2010-01-07) + + - workaround: subtle extraction bug (fix forthcoming) + + Version 0.1.1 (2010-01-06) + + - initial public release + + Bugs ---- Please report problems via the BaculaFS issue tracking system: `<http://code.google.com/p/baculafs/issues/list>`_ - LICENSE + Credits + ------- + + BaculaFS contains SQL queries that were adapted from Bacula, + Copyright \|(C)\| 2000-2010 Free Software Foundation Europe e.V. + + License ------- BaculaFS is free software: you can redistribute it and/or modify @@ -361,11 +542,14 @@ along with this program. If not, see `<http://www.gnu.org/licenses/>`_. + .. [*] Bacula is a registered trademark of Kern Sibbald. + .. \|(C)\| unicode:: 0xA9 .. copyright sign Platform: Linux Classifier: Development Status :: 3 - Alpha +Classifier: Topic :: System :: Filesystems Classifier: Topic :: System :: Archiving :: Backup Classifier: Intended Audience :: System Administrators Classifier: Environment :: No Input/Output (Daemon)
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/BaculaFS.egg-info/SOURCES.txt ^
@@ -16,4 +16,5 @@ baculafs/LogFile.py baculafs/SQL.py baculafs/__init__.py -baculafs/baculafs \ No newline at end of file +baculafs/baculafs +baculafs/junk.py \ No newline at end of file
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/PKG-INFO ^
@@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: BaculaFS -Version: 0.1.0 +Version: 0.1.7 Summary: Bacula Filesystem in USErspace Home-page: http://code.google.com/p/baculafs Author: Avi Rozen @@ -10,64 +10,67 @@ BaculaFS ======== - BaculaFS_ - Exposes the Bacula_ catalog and storage as a Filesystem in + BaculaFS_ - Exposes the Bacula_ []_ catalog and storage as a Filesystem in USErspace (FUSE_). .. _BaculaFS: http://code.google.com/p/baculafs .. _Bacula: http://www.bacula.org .. _FUSE: http://fuse.sourceforge.net/ - Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> + Copyright \|(C)\| 2009, 2010 Avi Rozen <avi.rozen@gmail.com> - INTRODUCTION + .. contents:: + + Introduction ------------ - BaculaFS* is a tool, independent of Bacula, that represents the - Bacula catalog and backup storage media as a read-only filesystem in - userspace. + BaculaFS is a tool, developed independently of Bacula, that + represents the Bacula catalog and backup storage media as a read-only + filesystem in userspace. BaculaFS is specifically designed to cater for the following use-cases: - maintaining a remote snapshot of the files in the backup storage - using `rsync`_ + using `rsync`_ or `duplicity`_ - auditing the contents of backup jobs, without resorting to SQL - queries + queries - comparing backup jobs (using several mount points) - Note that BaculaFS is a maintenance tool - it's operation may + Note that BaculaFS is a maintenance tool - its operation may interfere with the normal operation of a live Bacula setup (see the - LIMITATIONS_ section below). + Limitations_ section below). .. _rsync: http://samba.anu.edu.au/rsync/ + .. _duplicity: http://www.nongnu.org/duplicity/ - REQUIREMENTS + Requirements ------------ BaculaFS has been tested with the following set of required software packages: - + Bacula 3.0.2 with one of the following database backends: - - * SQLite_ 3.6.21 - * MySQL_ 5.4.1 - * PostgreSQL_ 8.3.9 + + Bacula 5.0.2 with one of the following database backends: + * SQLite_ 3.7.3 + * MySQL_ 5.1.49 + * PostgreSQL_ 8.4.5 + + FUSE_ components: - * Python FUSE 0.2 - * FUSE library 2.8.1 - * fusermount 2.8.1 - * FUSE kernel interface 7.12 - - + Python_ 2.5.4, with the following additional libraries: - - * MySQLdb_ 1.2.2 - * psycopg2_ 2.0.13 - * pexpect_ 2.3 - + * Python FUSE 0.2.1 + * FUSE library 2.8.4 + * fusermount 2.8.4 + * FUSE kernel interface 7.12 + + + Python_ 2.6.6, with the following additional libraries: + + * MySQLdb_ 1.2.2 + * psycopg2_ 2.2.1 + * pexpect_ 2.3 + + attr_ extended attributes utilities 2.4.44 BaculaFS requires the following in order to function: @@ -75,7 +78,7 @@ + access to the database that's used to store the Bacula catalog + access to the Bacula storage device + access to the Bacula ``bextract`` utility (bundled with the Bacula - storage daemon installation) + storage daemon installation) .. _SQLite: http://www.sqlite.org/ .. _MySQL: http://www.mysql.com/ @@ -88,7 +91,7 @@ .. _attr: http://savannah.nongnu.org/projects/attr - INSTALLATION + Installation ------------ Extract the source code archive to a temporary directory, ``cd`` to @@ -96,117 +99,136 @@ :: - python setup.py install + python setup.py install - USAGE + Usage ----- :: - baculafs [mountpoint] [options] - - Options: - --version show program's version number and exit - -h, --help show this help message and exit - -o opt,[opt...] mount options - -o driver=mysql\|postgresql\|sqlite\|sqlite3 - database driver [default: sqlite3] - -o host=HOST database server address [default: localhost] - -o port=PORT database server port - -o database=PATH database name [default: bacula] - -o username=USERNAME database user name [default: bacula] - -o password=PASSWORD database password - -o conf=PATH storage daemon configuration file [default: - /etc/bacula/bacula-sd.conf] - -o client=CLIENT file daemon name - -o fileset=FILESET backup fileset - -o device=DEVICE storage device name [default: FileStorage] - -o datetime='YYYY-MM-DD hh:mm:ss' - snapshot date/time [default: now] - -o recent_job select contents of most recent job only [default: - False] - -o joblist='JOBID1 JOBID2 ...' - select contents of specified list of jobs - -o cleanup clean cache directory upon umount [default: False] - -o move_root make absolute path symlinks point to path under - mount point [default: False] - -o prefetch_attrs read and parse attributes for all files upon - filesystem initialization [default: False] - -o prefetch_symlinks extract all symbolic links upon filesystem - initialization (implies prefetch_attrs) [default: - False] - -o prefetch_regex=REGEX - extract all objects that match REGEX upon - filesystem initialization (implies prefetch_attrs) - -o prefetch_recent extract contents of most recent non-full job upon - filesystem initialization (implies - prefetch_symlinks) [default: False] - -o prefetch_diff=PATH extract files that do not match files at PATH - (hint: speeds up rsync; implies prefetch_symlinks) - -o prefetch_everything - extract everything upon filesystem initialization - (complete restore to cache) [default: False] - -o user_cache_path=PATH - user specified cache path (hint: combine this with - one of the prefetch options) [default: none] - -o logging=debug\|info\|warning\|critical\|error - logging level [default: warning] - -o syslog log to both syslog and console [default: False] - - FUSE options: - -d -o debug enable debug output (implies -f) - -f foreground operation - -s disable multi-threaded operation - - -o allow_other allow access to other users - -o allow_root allow access to root - -o nonempty allow mounts over non-empty file/dir - -o default_permissions enable permission checking by kernel - -o fsname=NAME set filesystem name - -o subtype=NAME set filesystem type - -o large_read issue large read requests (2.4 only) - -o max_read=N set maximum size of read requests - - -o hard_remove immediate removal (don't hide files) - -o use_ino let filesystem set inode numbers - -o readdir_ino try to fill in d_ino in readdir - -o direct_io use direct I/O - -o kernel_cache cache files in kernel - -o [no]auto_cache enable caching based on modification times (off) - -o umask=M set file permissions (octal) - -o uid=N set file owner - -o gid=N set file group - -o entry_timeout=T cache timeout for names (1.0s) - -o negative_timeout=T cache timeout for deleted names (0.0s) - -o attr_timeout=T cache timeout for attributes (1.0s) - -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout) - -o intr allow requests to be interrupted - -o intr_signal=NUM signal to send on interrupt (10) - -o modules=M1[:M2...] names of modules to push onto filesystem stack - - -o max_write=N set maximum size of write requests - -o max_readahead=N set maximum readahead - -o async_read perform reads asynchronously (default) - -o sync_read perform reads synchronously - -o atomic_o_trunc enable atomic open+truncate support - -o big_writes enable larger than 4kB writes - -o no_remote_lock disable remote file locking - - Module options: - - [subdir] - -o subdir=DIR prepend this directory to all paths (mandatory) - -o [no]rellinks transform absolute symlinks to relative - - [iconv] - -o from_code=CHARSET original encoding of file names (default: UTF-8) - -o to_code=CHARSET new encoding of the file names (default: UTF-8) - + baculafs [mountpoint] [options] + + Options: + --version show program's version number and exit + -h, --help show this help message and exit + -o opt,[opt...] mount options + -o driver=mysql\|postgresql\|sqlite3 + database driver [default: sqlite3] + -o host=HOST database server address [default: localhost] + -o port=PORT database server port + -o database=PATH database name [default: bacula] + -o username=USERNAME database user name [default: bacula] + -o password=PASSWORD database password (use '-o password= ' to get a + password prompt; if not provided, the password is + read from the DATABASE_PASSWORD environment + variable) + -o conf=PATH storage daemon configuration file [default: + /etc/bacula/bacula-sd.conf] + -o client=CLIENT file daemon name + -o fileset=FILESET backup fileset + -o device=DEVICE storage device name [default: FileStorage] + -o datetime='YYYY-MM-DD hh:mm:ss' + snapshot date/time [default: now] + -o recent_job select contents of most recent job only [default: + False] + -o joblist='JOBID1 JOBID2 ...' + select contents of specified list of jobs + -o cleanup clean cache directory upon umount [default: False] + -o move_root make absolute path symlinks point to path under + mount point [default: False] + -o prefetch_attrs read and parse attributes for all files upon + filesystem initialization [default: False] + -o prefetch_symlinks extract all symbolic links upon filesystem + initialization (implies prefetch_attrs) [default: + False] + -o prefetch_regex=REGEX + extract all objects that match REGEX upon + filesystem initialization (implies prefetch_attrs) + -o prefetch_recent extract contents of most recent non-full job upon + filesystem initialization (implies + prefetch_symlinks) [default: False] + -o prefetch_diff=PATH extract files that do not match files at PATH + (hint: speeds up rsync; implies prefetch_symlinks) + -o prefetch_difflist=DIFFLIST + extract files that do not match files in DIFFLIST + (list line format: 'Day Mon DD hh:mm:ss YYYY PATH'; + use '-' to read from standard input; hint: format + matches output of 'duplicity list-current-files -v0 + target_url'; implies prefetch_symlinks) + -o prefetch_list=LIST extract files that match files in LIST (list should + contains one absolute file path per line; use '-' + to read from standard input; implies + prefetch_symlinks) + -o prefetch_everything + extract everything upon filesystem initialization + (complete restore to cache) [default: False] + -o batch_list list files to be prefetched and exit [default: + False] + -o batch_bsr dump contnets of bsr file for extracting prefetched + files and exit [default: False] + -o batch_extract extract prefetched files to mount point and exit + [default: False] + -o user_cache_path=PATH + user specified cache path (hint: combine this with + one of the prefetch options) [default: none] + -o logging=debug\|info\|warning\|critical\|error + logging level [default: info] + -o syslog log to both syslog and console [default: False] + + FUSE options: + -d -o debug enable debug output (implies -f) + -f foreground operation + -s disable multi-threaded operation + + -o allow_other allow access to other users + -o allow_root allow access to root + -o nonempty allow mounts over non-empty file/dir + -o default_permissions enable permission checking by kernel + -o fsname=NAME set filesystem name + -o subtype=NAME set filesystem type + -o large_read issue large read requests (2.4 only) + -o max_read=N set maximum size of read requests + + -o hard_remove immediate removal (don't hide files) + -o use_ino let filesystem set inode numbers + -o readdir_ino try to fill in d_ino in readdir + -o direct_io use direct I/O + -o kernel_cache cache files in kernel + -o [no]auto_cache enable caching based on modification times (off) + -o umask=M set file permissions (octal) + -o uid=N set file owner + -o gid=N set file group + -o entry_timeout=T cache timeout for names (1.0s) + -o negative_timeout=T cache timeout for deleted names (0.0s) + -o attr_timeout=T cache timeout for attributes (1.0s) + -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout) + -o intr allow requests to be interrupted + -o intr_signal=NUM signal to send on interrupt (10) + -o modules=M1[:M2...] names of modules to push onto filesystem stack + + -o max_write=N set maximum size of write requests + -o max_readahead=N set maximum readahead + -o async_read perform reads asynchronously (default) + -o sync_read perform reads synchronously + -o atomic_o_trunc enable atomic open+truncate support + -o big_writes enable larger than 4kB writes + -o no_remote_lock disable remote file locking + + Module options: + + [subdir] + -o subdir=DIR prepend this directory to all paths (mandatory) + -o [no]rellinks transform absolute symlinks to relative + + [iconv] + -o from_code=CHARSET original encoding of file names (default: UTF-8) + -o to_code=CHARSET new encoding of the file names (default: UTF-8) + - OPERATION + Operation --------- - INITIALIZATION + Initialization ~~~~~~~~~~~~~~ BaculaFS starts by running several SQL queries against the Bacula @@ -223,7 +245,7 @@ At this point the filesystem is ready. - CACHE + Cache ~~~~~ Opening a file for reading causes BaculaFS to run ``bextract`` in @@ -239,9 +261,9 @@ For example: - use ``-o prefetch_attrs`` for storage space usage analysis - (e.g. with Baobab_) + (e.g. with Baobab_) - use ``-o prefetch_symlinks`` for any manual filesystem traversal - with command line or GUI tools (``find``, ``mc``, etc.) + with command line or GUI tools (``find``, ``mc``, etc.) - use ``-o prefetch_diff`` with ``rsync`` .. _Baobab: http://www.marzocca.net/linux/baobab/ @@ -250,7 +272,24 @@ filesystem, with ``-o cleanup``. It may also be reused between mount operations with ``-o user_cache_path``. - EXTENDED ATTRIBUTES + Batch Mode + ~~~~~~~~~~ + + BaculaFS may be used in batch mode with ``-o batch_extract`` in + order to extract files from the Bacula storage device and then exit + without mounting the filesystem. The mountpoint specified at the + command line is then treated as the destination directory for + extracted files. + + The list of files, to be extracted in batch mode, is determined by the + various cache prefetch options, and may be dumped with ``-o + batch_list``. + + The bootstrap file that is generated, in order to extract the files, + can also be dumped to standard output with ``-o batch_bsr``. + + + Extended Attributes ~~~~~~~~~~~~~~~~~~~ BaculaFS uses extended file attributes to expose Bacula specific @@ -259,35 +298,39 @@ :: - user.baculafs.FileIndex - user.baculafs.JobId - user.baculafs.LStat - user.baculafs.MD5 + user.baculafs.FileIndex + user.baculafs.JobId + user.baculafs.LStat + user.baculafs.MD5 + + Note that ``user.baculafs.MD5`` shows whatever digest Bacula was + configured to calculate for the file, be it MD5, SHA1, SHA256 or + SHA512. The root directory has several more attributes, that expose filesystem instance-specific information: :: - user.baculafs.cache_prefix - user.baculafs.client - user.baculafs.datetime - user.baculafs.fileset - user.baculafs.joblist + user.baculafs.cache_prefix + user.baculafs.client + user.baculafs.datetime + user.baculafs.fileset + user.baculafs.joblist and several more attributes for monitoring the file extraction process: - + :: - user.baculafs.bextract.failures - user.baculafs.bextract.path - user.baculafs.bextract.pending - user.baculafs.bextract.retries - user.baculafs.bextract.state - user.baculafs.bextract.volume + user.baculafs.bextract.failures + user.baculafs.bextract.path + user.baculafs.bextract.pending + user.baculafs.bextract.retries + user.baculafs.bextract.state + user.baculafs.bextract.volume - MISSING VOLUMES + Missing Volumes ~~~~~~~~~~~~~~~ If the storage device is a tape drive then it's possible that @@ -301,16 +344,98 @@ :: - attr -s baculafs.bextract.state -V run <mount-point> + attr -s baculafs.bextract.state -V run <mount-point> Please note that this feature has undergone only rudimentary testing. Expect breakage. - LIMITATIONS + Examples + ~~~~~~~~ + + Mount the most recent backup snapshot for Bacula client ``client-fd`` + and fileset ``client-fileset``; SQLite database backend (default): + + :: + + baculafs -o client=client-fd,fileset=client-fileset /path/to/mount/point + + Mount the contents of the specified list of backup jobs; MySQL database + backend; prompt for password and prefetch to cache all symbolic links: + + :: + + baculafs -o joblist='1001 1003',client=client-fd,fileset=client-fileset \ + -o driver=mysql,password=,prefetch_symlinks /path/to/mount/point + + Mount the contents of the most recent backup job only: + + :: + + baculafs -o recent_job,client=client-fd,fileset=client-fileset /path/to/mount/point + + Mount the contents of the fileset snapshot before the specified + date/time: + + :: + + baculafs -o datetime='2009-05-23 00:00:00' \ + -o client=client-fd,fileset=client-fileset /path/to/mount/point + + Allow other users to access filesystem, set logging level to ``debug`` + and stay in foreground, so that ``bextract`` messages may be examined; + assume a single Bacula fileset is defined for client ``client-fd``: + + :: + + baculafs -f -o allow_other,client=client-fd,logging=debug /path/to/mount/point + + Prefetch to cache files that do not exist, or whose modification time + or size differ from those at the specified directory (i.e. files that + would be sent to that directory using ``rsync``); and cleanup cache after + un-mounting: + + :: + + baculafs -o prefetch_diff=/path/to/rsync/target,cleanup \ + -o client=client-fd /path/to/mount/point + + Batch update a snapshot of the current files in backup (the filesystem + is not mounted): + + :: + + baculafs -o batch_extract,prefetch_diff=/path/to/snapshot,cleanup \ + -o client=client-fd /path/to/snapshot/ + + (this is not as accurate as mounting the filesystem, like in the + previous example, and then updating the snapshot with ``rsync`` - but + it is faster). + + + Limitations ----------- - LOCKING + Common Problems + ~~~~~~~~~~~~~~~ + + BaculaFS will abort if no backup job was run for the + client/fileset combination selected by the user. + + BaculaFS will abort if the target mount point is in use. Note, + however, that mount point availability is checked by FUSE only after + the (potentially lengthy) initialization of BaculaFS. + + Browsing a BaculaFS mount point with a file manager like Nautilus_ + is liable to be very slow. This is because the file manager reads data + from each file being listed, in order to determine its type, generate + a thumbnail etc. This, in turn, triggers BaculaFS to extract the + files, one at a time, from the Bacula storage device to its cache + directory. + + .. _Nautilus: http://live.gnome.org/Nautilus + + Locking ~~~~~~~ Access to the storage device by different instances of BaculaFS is @@ -323,28 +448,84 @@ should note: - the lock is advisory, meaning that it does not prevent the Bacula - storage daemon itself from accessing the storage device while in use - by BaculaFS + storage daemon itself from accessing the storage device while in use + by BaculaFS - depending on your setup, the lock may not work if the storage daemon - configuration file is accessed via NFS + configuration file is accessed via NFS + + + Unsupported Features + ~~~~~~~~~~~~~~~~~~~~ + + BaculaFS depends on ``bextract`` and thus inherits its + limitations: + + - BaculaFS can be used with Windows filesets, but it does not + reproduce any Windows specific file attributes + - Encrypted backup files are not supported + - BaculaFS must be started with enough permissions (typically as + ``root``) in order to allow ``bextract`` to extract files from the + Bacula storage + Changelog + --------- + Version 0.1.7 (2010-12-30) + + - fixed: compatibility issues with bacula v2.4.4 and FUSE 7.8 + - fixed: synthesize missing inode numbers with -o use_ino + - modified: decode value of user.baculafs.MD5 extended file attribute + - added: cache prefetch specified list of files + - added: changelog to README + - added: batch extract mode + + Version 0.1.6 (2010-09-19) + + - fixed cache prefetch by regex + + Version 0.1.5 (2010-07-06) + + - fixed: removed reference to obsolete db field Copy + - fixed: recent_job option with MySQL + - added: read database password from environment variable DATABASE_PASSWORD + - added: usage examples to README + + Version 0.1.4 (2010-02-07) + + - added: cache prefetch based on duplicity file listing + Version 0.1.3 (2010-01-13) - WINDOWS FILESETS - ~~~~~~~~~~~~~~~~ + - fixed: (again) prefetch restore of files split between volumes + - fixed: missing import sys + - fixed: spurious linebreaks in debug log - BaculaFS can be used with Windows backup filesets, but it does not - reproduce any Windows specific file attributes. This is because - ``bextract`` does not extract Windows specific file attributes on - Linux. + Version 0.1.2 (2010-01-13) - BUGS + - fixed: prefetch_recent when joblist contains a single non full job + - fixed: prefetch restore of files split between volumes + - added: copyright, trademark and license blurbs + + Version 0.1.1 (2010-01-07) + + - workaround: subtle extraction bug (fix forthcoming) + + Version 0.1.1 (2010-01-06) + + - initial public release + + Bugs ---- Please report problems via the BaculaFS issue tracking system: `<http://code.google.com/p/baculafs/issues/list>`_ - LICENSE + Credits + ------- + + BaculaFS contains SQL queries that were adapted from Bacula, + Copyright \|(C)\| 2000-2010 Free Software Foundation Europe e.V. + + License ------- BaculaFS is free software: you can redistribute it and/or modify @@ -361,11 +542,14 @@ along with this program. If not, see `<http://www.gnu.org/licenses/>`_. + .. [*] Bacula is a registered trademark of Kern Sibbald. + .. \|(C)\| unicode:: 0xA9 .. copyright sign Platform: Linux Classifier: Development Status :: 3 - Alpha +Classifier: Topic :: System :: Filesystems Classifier: Topic :: System :: Archiving :: Backup Classifier: Intended Audience :: System Administrators Classifier: Environment :: No Input/Output (Daemon)
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/README.rst ^
@@ -2,63 +2,66 @@ BaculaFS ======== -BaculaFS_ - Exposes the Bacula_ catalog and storage as a Filesystem in +BaculaFS_ - Exposes the Bacula_ []_ catalog and storage as a Filesystem in USErspace (FUSE_). .. _BaculaFS: http://code.google.com/p/baculafs .. _Bacula: http://www.bacula.org .. _FUSE: http://fuse.sourceforge.net/ -Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +Copyright \|(C)\| 2009, 2010 Avi Rozen <avi.rozen@gmail.com> -INTRODUCTION +.. contents:: + +Introduction ------------ -BaculaFS* is a tool, independent of Bacula, that represents the -Bacula catalog and backup storage media as a read-only filesystem in -userspace. +BaculaFS is a tool, developed independently of Bacula, that +represents the Bacula catalog and backup storage media as a read-only +filesystem in userspace. BaculaFS is specifically designed to cater for the following use-cases: - maintaining a remote snapshot of the files in the backup storage - using `rsync`_ + using `rsync`_ or `duplicity`_ - auditing the contents of backup jobs, without resorting to SQL queries - comparing backup jobs (using several mount points) -Note that BaculaFS is a maintenance tool - it's operation may +Note that BaculaFS is a maintenance tool - its operation may interfere with the normal operation of a live Bacula setup (see the -LIMITATIONS_ section below). +Limitations_ section below). .. _rsync: http://samba.anu.edu.au/rsync/ +.. _duplicity: http://www.nongnu.org/duplicity/ -REQUIREMENTS +Requirements ------------ BaculaFS has been tested with the following set of required software packages: -+ Bacula 3.0.2 with one of the following database backends: ++ Bacula 5.0.2 with one of the following database backends: - * SQLite_ 3.6.21 - * MySQL_ 5.4.1 - * PostgreSQL_ 8.3.9 + * SQLite_ 3.7.3 + * MySQL_ 5.1.49 + * PostgreSQL_ 8.4.5 + FUSE_ components: - * Python FUSE 0.2 - * FUSE library 2.8.1 - * fusermount 2.8.1 + * Python FUSE 0.2.1 + * FUSE library 2.8.4 + * fusermount 2.8.4 * FUSE kernel interface 7.12 -+ Python_ 2.5.4, with the following additional libraries: ++ Python_ 2.6.6, with the following additional libraries: * MySQLdb_ 1.2.2 - * psycopg2_ 2.0.13 - * pexpect_ 2.3 + * psycopg2_ 2.2.1 + * pexpect_ 2.3 + attr_ extended attributes utilities 2.4.44 @@ -80,7 +83,7 @@ .. _attr: http://savannah.nongnu.org/projects/attr -INSTALLATION +Installation ------------ Extract the source code archive to a temporary directory, ``cd`` to @@ -90,7 +93,7 @@ python setup.py install -USAGE +Usage ----- :: @@ -101,13 +104,16 @@ --version show program's version number and exit -h, --help show this help message and exit -o opt,[opt...] mount options - -o driver=mysql\|postgresql\|sqlite\|sqlite3 + -o driver=mysql\|postgresql\|sqlite3 database driver [default: sqlite3] -o host=HOST database server address [default: localhost] -o port=PORT database server port -o database=PATH database name [default: bacula] -o username=USERNAME database user name [default: bacula] - -o password=PASSWORD database password + -o password=PASSWORD database password (use '-o password= ' to get a + password prompt; if not provided, the password is + read from the DATABASE_PASSWORD environment + variable) -o conf=PATH storage daemon configuration file [default: /etc/bacula/bacula-sd.conf] -o client=CLIENT file daemon name @@ -135,14 +141,30 @@ prefetch_symlinks) [default: False] -o prefetch_diff=PATH extract files that do not match files at PATH (hint: speeds up rsync; implies prefetch_symlinks) + -o prefetch_difflist=DIFFLIST + extract files that do not match files in DIFFLIST + (list line format: 'Day Mon DD hh:mm:ss YYYY PATH'; + use '-' to read from standard input; hint: format + matches output of 'duplicity list-current-files -v0 + target_url'; implies prefetch_symlinks) + -o prefetch_list=LIST extract files that match files in LIST (list should + contains one absolute file path per line; use '-' + to read from standard input; implies + prefetch_symlinks) -o prefetch_everything extract everything upon filesystem initialization (complete restore to cache) [default: False] + -o batch_list list files to be prefetched and exit [default: + False] + -o batch_bsr dump contnets of bsr file for extracting prefetched + files and exit [default: False] + -o batch_extract extract prefetched files to mount point and exit + [default: False] -o user_cache_path=PATH user specified cache path (hint: combine this with one of the prefetch options) [default: none] -o logging=debug\|info\|warning\|critical\|error - logging level [default: warning] + logging level [default: info] -o syslog log to both syslog and console [default: False] FUSE options: @@ -195,10 +217,10 @@ -o to_code=CHARSET new encoding of the file names (default: UTF-8) -OPERATION +Operation --------- -INITIALIZATION +Initialization ~~~~~~~~~~~~~~ BaculaFS starts by running several SQL queries against the Bacula @@ -215,7 +237,7 @@ At this point the filesystem is ready. -CACHE +Cache ~~~~~ Opening a file for reading causes BaculaFS to run ``bextract`` in @@ -242,7 +264,24 @@ filesystem, with ``-o cleanup``. It may also be reused between mount operations with ``-o user_cache_path``. -EXTENDED ATTRIBUTES +Batch Mode +~~~~~~~~~~ + +BaculaFS may be used in batch mode with ``-o batch_extract`` in +order to extract files from the Bacula storage device and then exit +without mounting the filesystem. The mountpoint specified at the +command line is then treated as the destination directory for +extracted files. + +The list of files, to be extracted in batch mode, is determined by the +various cache prefetch options, and may be dumped with ``-o +batch_list``. + +The bootstrap file that is generated, in order to extract the files, +can also be dumped to standard output with ``-o batch_bsr``. + + +Extended Attributes ~~~~~~~~~~~~~~~~~~~ BaculaFS uses extended file attributes to expose Bacula specific @@ -256,6 +295,10 @@ user.baculafs.LStat user.baculafs.MD5 +Note that ``user.baculafs.MD5`` shows whatever digest Bacula was +configured to calculate for the file, be it MD5, SHA1, SHA256 or +SHA512. + The root directory has several more attributes, that expose filesystem instance-specific information: @@ -279,7 +322,7 @@ user.baculafs.bextract.state user.baculafs.bextract.volume -MISSING VOLUMES +Missing Volumes ~~~~~~~~~~~~~~~ If the storage device is a tape drive then it's possible that @@ -299,10 +342,92 @@ testing. Expect breakage. -LIMITATIONS +Examples +~~~~~~~~ + +Mount the most recent backup snapshot for Bacula client ``client-fd`` +and fileset ``client-fileset``; SQLite database backend (default): + +:: + + baculafs -o client=client-fd,fileset=client-fileset /path/to/mount/point + +Mount the contents of the specified list of backup jobs; MySQL database +backend; prompt for password and prefetch to cache all symbolic links: + +:: + + baculafs -o joblist='1001 1003',client=client-fd,fileset=client-fileset \ + -o driver=mysql,password=,prefetch_symlinks /path/to/mount/point + +Mount the contents of the most recent backup job only: + +:: + + baculafs -o recent_job,client=client-fd,fileset=client-fileset /path/to/mount/point + +Mount the contents of the fileset snapshot before the specified +date/time: + +:: + + baculafs -o datetime='2009-05-23 00:00:00' \ + -o client=client-fd,fileset=client-fileset /path/to/mount/point + +Allow other users to access filesystem, set logging level to ``debug`` +and stay in foreground, so that ``bextract`` messages may be examined; +assume a single Bacula fileset is defined for client ``client-fd``: + +:: + + baculafs -f -o allow_other,client=client-fd,logging=debug /path/to/mount/point + +Prefetch to cache files that do not exist, or whose modification time +or size differ from those at the specified directory (i.e. files that +would be sent to that directory using ``rsync``); and cleanup cache after +un-mounting: + +:: + + baculafs -o prefetch_diff=/path/to/rsync/target,cleanup \ + -o client=client-fd /path/to/mount/point + +Batch update a snapshot of the current files in backup (the filesystem +is not mounted): + +:: + + baculafs -o batch_extract,prefetch_diff=/path/to/snapshot,cleanup \ + -o client=client-fd /path/to/snapshot/ + +(this is not as accurate as mounting the filesystem, like in the +previous example, and then updating the snapshot with ``rsync`` - but +it is faster). + + +Limitations ----------- -LOCKING +Common Problems +~~~~~~~~~~~~~~~ + +BaculaFS will abort if no backup job was run for the +client/fileset combination selected by the user. + +BaculaFS will abort if the target mount point is in use. Note, +however, that mount point availability is checked by FUSE only after +the (potentially lengthy) initialization of BaculaFS. + +Browsing a BaculaFS mount point with a file manager like Nautilus_ +is liable to be very slow. This is because the file manager reads data +from each file being listed, in order to determine its type, generate +a thumbnail etc. This, in turn, triggers BaculaFS to extract the +files, one at a time, from the Bacula storage device to its cache +directory. + +.. _Nautilus: http://live.gnome.org/Nautilus + +Locking ~~~~~~~ Access to the storage device by different instances of BaculaFS is @@ -321,22 +446,78 @@ configuration file is accessed via NFS +Unsupported Features +~~~~~~~~~~~~~~~~~~~~ + +BaculaFS depends on ``bextract`` and thus inherits its +limitations: + +- BaculaFS can be used with Windows filesets, but it does not + reproduce any Windows specific file attributes +- Encrypted backup files are not supported +- BaculaFS must be started with enough permissions (typically as + ``root``) in order to allow ``bextract`` to extract files from the + Bacula storage + +Changelog +--------- +Version 0.1.7 (2010-12-30) + +- fixed: compatibility issues with bacula v2.4.4 and FUSE 7.8 +- fixed: synthesize missing inode numbers with -o use_ino +- modified: decode value of user.baculafs.MD5 extended file attribute +- added: cache prefetch specified list of files +- added: changelog to README +- added: batch extract mode -WINDOWS FILESETS -~~~~~~~~~~~~~~~~ +Version 0.1.6 (2010-09-19) -BaculaFS can be used with Windows backup filesets, but it does not -reproduce any Windows specific file attributes. This is because -``bextract`` does not extract Windows specific file attributes on -Linux. +- fixed cache prefetch by regex -BUGS +Version 0.1.5 (2010-07-06) + +- fixed: removed reference to obsolete db field Copy +- fixed: recent_job option with MySQL +- added: read database password from environment variable DATABASE_PASSWORD +- added: usage examples to README + +Version 0.1.4 (2010-02-07) + +- added: cache prefetch based on duplicity file listing + +Version 0.1.3 (2010-01-13) + +- fixed: (again) prefetch restore of files split between volumes +- fixed: missing import sys +- fixed: spurious linebreaks in debug log + +Version 0.1.2 (2010-01-13) + +- fixed: prefetch_recent when joblist contains a single non full job +- fixed: prefetch restore of files split between volumes +- added: copyright, trademark and license blurbs + +Version 0.1.1 (2010-01-07) + +- workaround: subtle extraction bug (fix forthcoming) + +Version 0.1.1 (2010-01-06) + +- initial public release + +Bugs ---- Please report problems via the BaculaFS issue tracking system: `<http://code.google.com/p/baculafs/issues/list>`_ -LICENSE +Credits +------- + +BaculaFS contains SQL queries that were adapted from Bacula, +Copyright \|(C)\| 2000-2010 Free Software Foundation Europe e.V. + +License ------- BaculaFS is free software: you can redistribute it and/or modify @@ -353,5 +534,7 @@ along with this program. If not, see `<http://www.gnu.org/licenses/>`_. +.. [*] Bacula is a registered trademark of Kern Sibbald. +.. \|(C)\| unicode:: 0xA9 .. copyright sign
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/baculafs/Base64.py ^
@@ -1,3 +1,27 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + class Base64 : ''' Bacula specific implementation of a base64 decoder
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/baculafs/Catalog.py ^
@@ -1,3 +1,27 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + from datetime import datetime from SQL import * @@ -74,7 +98,7 @@ diff_jobs = self.db.query(SQL.temp) self.db.query(SQL.incr_jobs_temp % (diff_jobs[-1][1], self.datetime, self.client_id, self.fileset[1]), fetch=False) jobs = self.db.query(SQL.jobs) - self.most_recent_jobid = jobs[-1][0] if len(jobs) > 1 else -1 + self.most_recent_jobid = jobs[-1][0] if jobs[-1][1] != 'F' else -1 # select files from the most recent job only if select_recent_job and self.most_recent_jobid > 0 : jobs = [jobs[-1]]
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/baculafs/Database.py ^
@@ -1,4 +1,29 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + import os +from getpass import getpass from Base64 import * from SQL import * @@ -8,12 +33,11 @@ This class shields the rest of the code from the pesky details of actually accessing one of the supported databases. ''' - drivers = [SQL.MYSQL, SQL.POSTGRESQL, SQL.SQLITE, SQL.SQLITE3] + drivers = [SQL.MYSQL, SQL.POSTGRESQL, SQL.SQLITE3] default_database = { SQL.MYSQL: 'bacula', SQL.POSTGRESQL: 'bacula', - SQL.SQLITE: '/var/lib/bacula/bacula.db', SQL.SQLITE3: '/var/lib/bacula/bacula.db' } @@ -28,18 +52,21 @@ self.driver = driver if not database : database = Database.default_database[self.driver] + # user supplied empty password -> prompt + if password == '' : + password = getpass() + # user supplied no password -> query environment or use empty string + if password == None : + try: + password = os.environ['DATABASE_PASSWORD'] + except KeyError: + password = '' if self.driver == SQL.MYSQL : from MySQLdb import connect self.connection = connect(host=host, port=port, user=username, passwd=password, db=database) elif self.driver == SQL.POSTGRESQL : from psycopg2 import connect self.connection = connect(host=host, port=port, user=username, password=password, database=database) - elif self.driver == SQL.SQLITE : - from sqlite import connect - database = os.path.expanduser(database) - if not os.path.isfile(database) or not os.access(database, os.R_OK) : - raise RuntimeError, 'cannot read from file %s' % database - self.connection = connect(database) elif self.driver == SQL.SQLITE3 : from sqlite3 import connect database = os.path.expanduser(database) @@ -64,10 +91,10 @@ def query(self, sql, fetch = True) : ''' - Execute SQL and fetch all results + Execute SQL and fetch all results as a list ''' self.logger.debug(sql) self.cursor.execute(sql) if fetch : - return self.cursor.fetchall() + return list(self.cursor.fetchall())
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/baculafs/FileSystem.py ^
@@ -1,6 +1,31 @@ -__version__ = '0.1.0' + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + +__version__ = '0.1.7' import os +import sys import stat import errno import copy @@ -10,10 +35,14 @@ import traceback import pexpect import fcntl +import time +import re +import binascii from LogFile import * from Database import * from Catalog import * +from SQL import * # pull in some spaghetti to make this stuff work without fuse-py being installed try: @@ -71,7 +100,18 @@ class FileSystem(Fuse) : - null_stat = fuse.Stat(st_mode = stat.S_IFDIR \| 0755, st_nlink = 2, st_ino = -1) + null_stat = fuse.Stat(st_mode=stat.S_IFDIR \| 0755, + st_ino=0, + st_dev=0, + st_nlink=2, + st_uid=0, + st_gid=0, + st_size=0, + st_atime=0, + st_mtime=0, + st_ctime=0, + st_blksize=0, + st_rdev=0) bacula_stat_fields = ['st_dev', 'st_ino', @@ -90,7 +130,7 @@ 'st_flags', 'st_streamid'] - fuse_stat_fields = dir(fuse.Stat()) + fuse_stat_fields = [attr for attr in dir(null_stat) if attr.startswith('st_')] xattr_prefix = 'user.baculafs.' xattr_fields = ['FileIndex', 'JobId', 'LStat', 'MD5'] @@ -115,14 +155,14 @@ self._initialized = False # default option values - self.logging = 'warning' + self.logging = 'info' self.syslog = False - self.driver = 'sqlite3' + self.driver = SQL.SQLITE3 self.database = None self.host = 'localhost' self.port = 0 self.username = 'bacula' - self.password = '' + self.password = None self.conf = '/etc/bacula/bacula-sd.conf' self.client = '' self.fileset = None @@ -130,6 +170,7 @@ self.datetime = None self.recent_job = False self.joblist = None + self.cache_prefix = None self.user_cache_path = None self.cleanup = False self.move_root = False @@ -138,7 +179,15 @@ self.prefetch_symlinks = False self.prefetch_recent = False self.prefetch_diff = None + self.prefetch_difflist = None + self.prefetch_list = None self.prefetch_everything = False + self.batch_mode = False + self.batch_list = False + self.batch_bsr = False + self.batch_extract = False + self.use_ino = False + self.max_ino = 0 self.dirs = { '/': { '': (FileSystem.null_stat,) } } self._bextract_status = copy.deepcopy(FileSystem.bextract_done) @@ -188,6 +237,19 @@ self.dirs[head][tail] = (FileSystem.null_stat,) self._add_parent_dirs(head) + def _update_inodes(self, head) : + ''' + generate unique st_ino for each missing st_ino + ''' + for tail in self.dirs[head] : + if self.dirs[head][tail][-1].st_ino == 0 : + if len(self.dirs[head][tail]) == 1: + self.dirs[head][tail] = (copy.deepcopy(FileSystem.null_stat),) + self.max_ino += 1 + self.dirs[head][tail][-1].st_ino = self.max_ino + subdir = '%s%s/' % (head, tail) + if subdir in self.dirs : + self._update_inodes(subdir) def _extract(self, path_list) : ''' @@ -227,9 +289,7 @@ rc, sig = self._bextract(items) # it seems that bextract does not restore mtime for symlinks # so we create a normal file with same mtime as stored symlink - # (note that we only use that file if the cache path was - # supplied by the user) - if rc == 0 : + if rc == 0 and not self.batch_mode : for item in items : if item[0] : symlinkfile = item[0][0] @@ -279,19 +339,14 @@ is_symlink = stat.S_ISLNK(bs.st_mode) found = False if os.path.exists(realpath) or os.path.lexists(realpath) : - # do not trust user supplied cache path: # make sure that stat info of realpath matches path - if self.user_cache_path : - s = os.lstat(realpath) - conds = [getattr(s, attr) == getattr(bs, attr) - for attr in ['st_mode', 'st_uid', 'st_gid', 'st_size', 'st_mtime']] - if is_symlink : - conds[-1] = (os.path.exists(symlinkpath) and - bs.st_mtime == os.stat(symlinkpath).st_mtime) - found = all(conds) - else : - found = True - if found : + s = os.lstat(realpath) + conds = [getattr(s, attr) == getattr(bs, attr) + for attr in ['st_mode', 'st_uid', 'st_gid', 'st_size', 'st_mtime']] + if is_symlink : + conds[-1] = (os.path.exists(symlinkpath) and + bs.st_mtime == os.stat(symlinkpath).st_mtime) + if all(conds) : return realpath, None, None # generate list of volumes for path fileindex, jobid = self.dirs[head][tail][0:2] @@ -300,11 +355,12 @@ volumes = [[volume[1], # 0-Volume volume[2], # 1-MediaType self.device, # 2-Device - jobs[0][1], # 3-VolSessionId - jobs[0][2], # 4-VolSessionTime - (volume[5] << 32) \| volume[7], # 5-VolAddr: StartAddr - (volume[6] << 32) \| volume[8], # 6-VolAddr: EndAddr - fileindex] # 7-FileIndex + jobs[0][0], # 3-JobId + jobs[0][1], # 4-VolSessionId + jobs[0][2], # 5-VolSessionTime + (volume[5] << 32) \| volume[7], # 6-VolAddr: StartAddr + (volume[6] << 32) \| volume[8], # 7-VolAddr: EndAddr + fileindex] # 8-FileIndex for volume in self.catalog.volumes if (volume[0] == jobid and volume[3] <= fileindex and @@ -369,8 +425,31 @@ ''' extract list of items from Bacula storage device ''' + if self.batch_list : + for item in items : + print item[1] + if (not self.batch_bsr and + not self.batch_extract) : + return (0, 0) + bsrpath = self._write_bsr(items) - cmd = 'bextract -b "%s" -c "%s" "%s" "%s"' % (bsrpath, self.conf, self.device, self.cache_path) + + if self.batch_bsr : + bsrfile = open(bsrpath, 'rt') + for line in bsrfile : + sys.stdout.write(line) + sys.stdout.flush() + bsrfile.close() + if not self.batch_extract : + return (0, 0) + + if self.batch_extract : + makedirs(self.fuse_args.mountpoint) + + cmd = 'bextract -b "%s" -c "%s" "%s" "%s"' % (bsrpath, + self.conf, + self.device, + self.cache_path if not self.batch_extract else self.fuse_args.mountpoint) self.logger.debug(cmd) self._bextract_set_status({'path': items[0][1], @@ -393,6 +472,7 @@ i = child.expect([self.fail_pattern, pexpect.EOF], timeout=None, searchwindowsize=200) + self.logfile.flush(flush_tail = True) if i == 0 : # count retries if missing == child.match.groups()[0] : @@ -434,21 +514,69 @@ self._bextract_increment_counter('failures', 1) return (child.exitstatus, child.signalstatus) + def _group_by_volume(self, items) : + ''' + return items grouped by volume + ''' + # group volumes + volumes = [] + for item in items : + for v in item[-1] : + found = False + findex = v[-1] + for vindex in xrange(0,len(volumes)) : + volume = volumes[vindex] + if not any(map(cmp, v[:-1], volume[:-1])) : + volume[-1].append(findex) + found = True + break + if not found : + volumes.append(v[:-1] + [[v[-1]]]) + + # compact list of file indices + for volume in volumes : + volume[-1] = list(set(volume[-1])) + volume[-1].sort() + l = len(volume[-1]) + findex = volume[-1][0] + findices = [(findex, findex)] + for idx in volume[-1][1:] : + next_idx = findices[-1][-1] + 1 + if idx == next_idx : + findices[-1] = (findices[-1][0], idx) + else : + findices.append((idx,idx)) + volume[-1] = findices + volume.append(l) + + # reorder volumes to ensure correct handling of + # files spanning multiple volumes + volumes.sort(cmp = lambda a,b : \ + (cmp(a[3],b[3]) or + cmp(a[8][-1][-1], b[8][0][0]))) + + return volumes + def _write_bsr(self, items) : ''' generate bsr for items to be extracted ''' bsrfd, bsrpath = tempfile.mkstemp(suffix='.bsr', dir=self.cache_bsrpath, text=True) - for item in items : - for volume in item[-1] : - os.write(bsrfd, 'Volume="%s"\n' % volume[0]) - os.write(bsrfd, 'MediaType="%s"\n' % volume[1]) - os.write(bsrfd, 'Device="%s"\n' % volume[2]) - os.write(bsrfd, 'VolSessionId=%d\n' % volume[3]) - os.write(bsrfd, 'VolSessionTime=%d\n' % volume[4]) - os.write(bsrfd, 'VolAddr=%d-%d\n' % (volume[5],volume[6])) - os.write(bsrfd, 'FileIndex=%d\n' % volume[7]) - os.write(bsrfd, 'Count=1\n') + volumes = self._group_by_volume(items) + for volume in volumes : + os.write(bsrfd, 'Volume="%s"\n' % volume[0]) + os.write(bsrfd, 'MediaType="%s"\n' % volume[1]) + os.write(bsrfd, 'Device="%s"\n' % volume[2]) + os.write(bsrfd, 'VolSessionId=%d\n' % volume[4]) + os.write(bsrfd, 'VolSessionTime=%d\n' % volume[5]) + if not self.bsr_compat : + os.write(bsrfd, 'VolAddr=%d-%d\n' % (volume[6],volume[7])) + for findex in volume[8] : + if findex[0] == findex[1] : + os.write(bsrfd, 'FileIndex=%d\n' % findex[0]) + else : + os.write(bsrfd, 'FileIndex=%d-%d\n' % findex) + os.write(bsrfd, 'Count=%d\n' % volume[9]) os.close(bsrfd) return bsrpath @@ -489,14 +617,22 @@ self.logger.warning(traceback.format_exc()) self.logfile = LogFile(self.logger, logging.DEBUG) - - def initialize(self): + def initialize(self, version) : ''' initialize database, catalog ''' self._setup_logging() + # batch mode + self.batch_mode = (self.batch_list or + self.batch_bsr or + self.batch_extract) + # disable INFO level logging in batch mode + if self.batch_mode and self.loglevel == logging.INFO : + self.loglevel = logging.WARNING + self.logger.setLevel(self.loglevel) + self.logger.info('Populating file system ... ') # setup cache @@ -511,6 +647,10 @@ self.cache_symlinks = os.path.normpath(self.cache_prefix + '/symlinks') makedirs(self.cache_symlinks) + # test for old version (2.x) of bacula + self.bsr_compat = int(version[0]) < 3 + if self.bsr_compat : + self.logger.debug('Detected old Bacula: %s' % version) # test access to sd conf file open(self.conf, 'r').close() # init bextract failure pattren @@ -534,12 +674,15 @@ self.db.close() prefetches = [] + difflist = {} # validate prefetch conditions if self.prefetch_everything : self.prefetch_recent = False self.prefetch_regex = None self.prefetch_diff = None + self.prefetch_difflist = None + self.prefetch_list = None self.prefetch_symlinks = True if self.prefetch_regex : try : @@ -560,10 +703,37 @@ # can't access target directory: show traceback and ignore self.logger.warning(traceback.format_exc()) self.prefetch_diff = None + if self.prefetch_difflist : + self.prefetch_difflist = os.path.normpath(os.path.expanduser(self.prefetch_difflist)) + try : + difflistfile = sys.stdin if self.prefetch_difflist == '-' else open(self.prefetch_difflist, 'rt') + for line in difflistfile.readlines(): + date = ' '.join(line.split()[:5]) + difflist[line[(len(date) + 1):].strip()] = time.strptime(date, '%a %b %d %H:%M:%S %Y') + difflistfile.close() + self.prefetch_symlinks = True + except : + # can't access/parse difflist: show traceback and ignore + self.logger.warning(traceback.format_exc()) + self.prefetch_difflist = None + if self.prefetch_list : + self.prefetch_list = os.path.normpath(os.path.expanduser(self.prefetch_list)) + try : + listfile = sys.stdin if self.prefetch_list == '-' else open(self.prefetch_list, 'rt') + matchlist = [line.strip() for line in listfile.readlines()] + listfile.close() + self.prefetch_symlinks = True + except : + # can't access/parse list: show traceback and ignore + self.logger.warning(traceback.format_exc()) + self.prefetch_list = None if self.prefetch_recent : self.prefetch_symlinks = True if self.prefetch_symlinks : self.prefetch_attrs = True + if 'use_ino' in self.fuse_args.optlist: + self.use_ino = True + self.prefetch_attrs = True # must figure out max st_ino for file in files : head = file[0] @@ -574,6 +744,10 @@ # make file entry if self.prefetch_attrs : entry = file[2:] + self._bacula_stat(file[-2]) + # find max st_ino + if self.use_ino: + if entry[-1].st_ino > self.max_ino : + self.max_ino = entry[-1].st_ino # detemine if we need to prefetch this entry filepath = head + tail if (not stat.S_ISDIR(entry[-1].st_mode) and @@ -581,9 +755,14 @@ (self.prefetch_recent and file[3] == self.catalog.most_recent_jobid) or (self.prefetch_regex and - regex.match(filepath)) or + regex.search(filepath)) or (self.prefetch_diff and - not self._match_stat(self.prefetch_diff + filepath, entry[-1])) or + not self._match_stat(self.prefetch_diff + filepath, entry[-1])) or + (self.prefetch_difflist and + (filepath[1:] not in difflist or + difflist[filepath[1:]][:-1] != time.localtime(entry[-1].st_mtime)[:-1])) or + (self.prefetch_list and + filepath in matchlist) or (self.prefetch_symlinks and stat.S_ISLNK(entry[-1].st_mode)))) : prefetches.append(filepath) @@ -600,13 +779,17 @@ # and finally self.dirs[head][tail] = entry + # fix st_ino + if self.use_ino: + self._update_inodes('/') + npf = len(prefetches) if npf > 0 : self.logger.info('Prefetching %d objects ... ' % npf) self._extract(prefetches) - self.logger.info('Cache directory is: %s' % self.cache_prefix) + self.logger.debug('Cache directory is: %s' % self.cache_prefix) self.joblist = ' '.join([str(job[0]) for job in self.catalog.jobs]) - self.logger.info('Jobs in file system: %s' % self.joblist) + self.logger.debug('Job ids in file system: %s' % self.joblist) self.logger.info('BaculaFS ready (%d files).' % len(files)) self._initialized = True @@ -615,7 +798,7 @@ ''' remove cache directory if required ''' - if self.cleanup and not self.user_cache_path : + if self.cleanup and not self.user_cache_path and self.cache_prefix : self.logger.info('removing cache directory: %s' % self.cache_prefix) shutil.rmtree(self.cache_prefix, ignore_errors = True) @@ -653,6 +836,9 @@ len(self.dirs[head][tail]) != 1 and n in FileSystem.xattr_fields) : val = str(self.dirs[head][tail][FileSystem.xattr_fields.index(n)]) + if n == 'MD5' and val != '0': + l = len(val) + val = binascii.b2a_hex(binascii.a2b_base64(val+'='((l3+8)/3-l)+'\n')) # padding # attribute not found if val == None : return -errno.ENODATA @@ -719,8 +905,11 @@ yield fuse.Direntry(key) for key in self.dirs[path].keys() : if len(key) > 0: - bs = self.getattr(path + key) - ino = bs.st_ino if bs.st_ino != 0 else -1 + if self.use_ino: + bs = self.getattr(path + key) + ino = bs.st_ino + else : + ino = 0 yield fuse.Direntry(key, ino=ino) def readlink(self, path): @@ -797,7 +986,7 @@ server.parser.add_option(mountopt="username", metavar="USERNAME", default=server.username, help="database user name [default: %default]") server.parser.add_option(mountopt="password", metavar="PASSWORD", default=server.password, - help="database password") + help="database password (use '-o password= ' to get a password prompt; if not provided, the password is read from the DATABASE_PASSWORD environment variable)") server.parser.add_option(mountopt="conf", metavar="PATH", default=server.conf, help="storage daemon configuration file [default: %default]") server.parser.add_option(mountopt="client", metavar="CLIENT", default=server.client, @@ -826,8 +1015,18 @@ help="extract contents of most recent non-full job upon filesystem initialization (implies prefetch_symlinks) [default: %default]") server.parser.add_option(mountopt="prefetch_diff", metavar="PATH", default=server.prefetch_diff, help="extract files that do not match files at PATH (hint: speeds up rsync; implies prefetch_symlinks)") + server.parser.add_option(mountopt="prefetch_difflist", metavar="DIFFLIST", default=server.prefetch_difflist, + help="extract files that do not match files in DIFFLIST (list line format: 'Day Mon DD hh:mm:ss YYYY PATH'; use '-' to read from standard input; hint: format matches output of 'duplicity list-current-files -v0 target_url'; implies prefetch_symlinks)") + server.parser.add_option(mountopt="prefetch_list", metavar="LIST", default=server.prefetch_list, + help="extract files that match files in LIST (list should contains one absolute file path per line; use '-' to read from standard input; implies prefetch_symlinks)") server.parser.add_option(mountopt="prefetch_everything", action="store_true", default=server.prefetch_everything, help="extract everything upon filesystem initialization (complete restore to cache) [default: %default]") + server.parser.add_option(mountopt="batch_list", action="store_true", default=server.batch_list, + help="list files to be prefetched and exit [default: %default]") + server.parser.add_option(mountopt="batch_bsr", action="store_true", default=server.batch_bsr, + help="dump contnets of bsr file for extracting prefetched files and exit [default: %default]") + server.parser.add_option(mountopt="batch_extract", action="store_true", default=server.batch_extract, + help="extract prefetched files to mount point and exit [default: %default]") server.parser.add_option(mountopt="user_cache_path", metavar="PATH", default=server.user_cache_path, help="user specified cache path (hint: combine this with one of the prefetch options) [default: %default]") server.parser.add_option(mountopt="logging", choices=LOGGING_LEVELS.keys(), metavar='\|'.join(LOGGING_LEVELS.keys()), default=server.logging, @@ -844,13 +1043,13 @@ # we initialize before main (i.e. not in fsinit) so that # any failure here aborts the mount try : - server.initialize() + server.initialize(bacula_version) except : - traceback.print_exc() server.shutdown() raise - server.main() + if not server.batch_mode : + server.main() # we shutdown after main, i.e. not in fsshutdown, because # calling fsshutdown with multithreaded==True seems to cause
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/baculafs/LogFile.py ^
@@ -1,3 +1,27 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + import logging import logging.handlers @@ -14,12 +38,25 @@ def __init__(self, logger, level) : self.logger = logger self.level = level + self.tail = '' def write(self, message) : - for line in message.splitlines() : + lines = (self.tail + message).splitlines() + for line in lines[:-1] : self.logger.log(self.level, line) + if message.endswith('\n') : + self.logger.log(self.level, lines[-1]) + self.tail = '' + elif len(lines) > 0 : + self.tail = lines[-1] + else : + self.tail = '' + - def flush(self) : + def flush(self, flush_tail = False) : + if flush_tail and self.tail : + self.logger.log(self.level, self.tail) + self.tail = '' for handler in self.logger.handlers : handler.flush()
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/baculafs/SQL.py ^
@@ -1,3 +1,27 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + class SQL : ''' Holds all SQL statements used by baculafs. @@ -6,7 +30,6 @@ MYSQL = 'mysql' POSTGRESQL = 'postgresql' - SQLITE = 'sqlite' SQLITE3 = 'sqlite3' clients = 'SELECT Client.Name,ClientId FROM Client' @@ -54,21 +77,6 @@ VolSessionTime INTEGER) ''', - SQLITE: ''' - CREATE TEMPORARY TABLE temp ( - JobId INTEGER UNSIGNED NOT NULL, - JobTDate BIGINT UNSIGNED, - ClientId INTEGER UNSIGNED, - Level CHAR, - JobFiles INTEGER UNSIGNED, - JobBytes BIGINT UNSIGNED, - StartTime TEXT, - VolumeName TEXT, - StartFile INTEGER UNSIGNED, - VolSessionId INTEGER UNSIGNED, - VolSessionTime INTEGER UNSIGNED) - ''', - SQLITE3: ''' CREATE TEMPORARY TABLE temp ( JobId INTEGER UNSIGNED NOT NULL, @@ -97,12 +105,6 @@ JobTDate BIGINT) ''', - SQLITE: ''' - CREATE TEMPORARY TABLE temp1 ( - JobId INTEGER UNSIGNED NOT NULL, - JobTDate BIGINT UNSIGNED) - ''', - SQLITE3: ''' CREATE TEMPORARY TABLE temp1 ( JobId INTEGER UNSIGNED NOT NULL, @@ -191,7 +193,7 @@ AND FileSet.FileSet='%s' ''' - jobs = 'SELECT DISTINCT JobId,StartTime FROM temp ORDER BY StartTime ASC' + jobs = 'SELECT DISTINCT JobId,Level,StartTime FROM temp ORDER BY StartTime ASC' base_jobs = ''' SELECT DISTINCT BaseJobId @@ -257,32 +259,6 @@ -- dummy comment for chomping extra parameter: %s ''', - SQLITE: ''' - SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId, - File.FilenameId AS FilenameId, LStat, MD5 - FROM Job, File, ( - SELECT MAX(JobTDate) AS JobTDate, PathId, FilenameId - FROM ( - SELECT JobTDate, PathId, FilenameId - FROM File JOIN Job USING (JobId) - WHERE File.JobId IN (%s) - UNION ALL - SELECT JobTDate, PathId, FilenameId - FROM BaseFiles - JOIN File USING (FileId) - JOIN Job ON (BaseJobId = Job.JobId) - WHERE BaseFiles.JobId IN (%s) - ) AS tmp GROUP BY PathId, FilenameId - ) AS T1 - WHERE (Job.JobId IN ( - SELECT DISTINCT BaseJobId FROM BaseFiles WHERE JobId IN (%s)) - OR Job.JobId IN (%s)) - AND T1.JobTDate = Job.JobTDate - AND Job.JobId = File.JobId - AND T1.PathId = File.PathId - AND T1.FilenameId = File.FilenameId - ''', - SQLITE3: ''' SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId, File.FilenameId AS FilenameId, LStat, MD5 @@ -313,13 +289,13 @@ SELECT JobId,VolSessionId,VolSessionTime, PoolId,StartTime,EndTime,JobFiles,JobBytes,JobTDate,Job,JobStatus, Type,Level,ClientId,Name,PriorJobId,RealEndTime,FileSetId, - SchedTime,RealEndTime,ReadBytes,HasBase + SchedTime,RealEndTime,HasBase FROM Job WHERE JobId IN (%s) ''' volumes = ''' SELECT JobMedia.JobId,VolumeName,MediaType,FirstIndex,LastIndex,StartFile, - JobMedia.EndFile,StartBlock,JobMedia.EndBlock,Copy, + JobMedia.EndFile,StartBlock,JobMedia.EndBlock, Slot,StorageId,InChanger FROM JobMedia,Media WHERE JobMedia.JobId IN (%s) AND JobMedia.MediaId=Media.MediaId ORDER BY JobMedia.JobId,VolIndex,JobMediaId
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/baculafs/__init__.py ^
@@ -1,7 +1,31 @@ + +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + """Expose the Bacula catalog as a FUSE file system. """ from FileSystem import __version__, main -__all__ = ["FileSystem", "Database", "Catalog", "Base64", "SQL"] +__all__ = ["FileSystem", "Database", "Catalog", "Base64", "SQL", "LogFile"]
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/baculafs/baculafs ^
@@ -1,5 +1,28 @@ #! /usr/bin/env python +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + '''BaculaFS mount script '''
[-] [+]	Added	BaculaFS-0.1.7.tar.gz/baculafs/junk.py ^
@@ -0,0 +1,6 @@ +import commands +baculafs_mount_points = [tokens[2] for tokens in + [line.split() for line in commands.getoutput('mount -v').split('\n')] + if tokens[4].endswith('baculafs')] +if self.fuse_args.mountpoint in baculafs_mount_points : + raise RuntimeError, 'mountpoint already in use by BaculaFS'
[-] [+]	Changed	BaculaFS-0.1.7.tar.gz/setup.py ^
@@ -1,5 +1,28 @@ #!/usr/bin/python +# BaculaFS - Bacula Filesystem in USErspace +# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com> +# +# BaculaFS contains SQL queries that were adapted from Bacula +# Copyright (C) 2000-2010 Free Software Foundation Europe e.V. +# +# This file is part of BaculaFS. +# +# BaculaFS is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Bacula is a registered trademark of Kern Sibbald. + from setuptools import setup, find_packages from baculafs import __version__ @@ -23,6 +46,7 @@ install_requires=['fuse-python>=0.2','pexpect>=2.3','MySQL-python>=1.2.2','psycopg2>=2.0.13'], classifiers = [ "Development Status :: 3 - Alpha", + "Topic :: System :: Filesystems", "Topic :: System :: Archiving :: Backup", "Intended Audience :: System Administrators", "Environment :: No Input/Output (Daemon)",

Changes of Revision 3