Help!

TAR & GZIP into multiple files

 
Post new topic   General Reply to Topic (not reply to a specific post)    Forums Home -> Setup RSS
Next:  POP3 debian email server problem  
Author Message
Christopher Ho
External


Since: Jul 12, 2007
Posts: 1



PostPosted: Thu Jul 12, 2007 7:06 pm    Post subject: TAR & GZIP into multiple files
Archived from groups: comp>os>linux>setup (more info?)

Hi all,

Is it possible to tar + gzip a huge directory and split into multiple
archives ?

The command I used to use is

cd /data/ (which has a directory "output" under it)

tar -czf /backup/data_May2007.tar.gz output

This output folder contains about a million files and after
compression the archive take the space about 15 Gb.

I want to separate the archive into DVD size but I don't want to use
up too much space to do all the operation.Any quick one line command I
can achieve it? Thanks.
Back to top
Douglas Mayne
External


Since: Dec 04, 2005
Posts: 206



PostPosted: Thu Jul 12, 2007 7:06 pm    Post subject: Re: TAR & GZIP into multiple files [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

On Thu, 12 Jul 2007 15:06:53 +0000, Christopher Ho wrote:

> Hi all,
>
> Is it possible to tar + gzip a huge directory and split into multiple
> archives ?
>
> The command I used to use is
>
> cd /data/ (which has a directory "output" under it)
>
> tar -czf /backup/data_May2007.tar.gz output
>
> This output folder contains about a million files and after
> compression the archive take the space about 15 Gb.
>
> I want to separate the archive into DVD size but I don't want to use
> up too much space to do all the operation.Any quick one line command I
> can achieve it? Thanks.
>
I am not sure about minimizing space used by potential intermediate files,
but one trick is to use the command split. man split.
Split breaks a larger file into blocks of the specified size.

The individual block files can be written to DVD. The blocks must be
pasted back together when restoring. Device mapper's "linear" module may
be useful in virtually pasting the components back together. Without that
trick, this is the basic idea:

$ cat /mnt/dvd/part.xxx >>restore.tgz

(For every block, and in sequence.)

Notes:
1. The entire archive may be unusable if any one DVD fails or becomes
unreadable.

2. Compute md5sums for each component written to DVD to provide a method
in the future of ensuring that the disc is still readable.

3. Consider adding an encryption layer to system backups.

4. As far as minimizing space, there are probably some tricks that can be
used to minimize intermediate files. These tricks would probably use
fifo's, loopbacks, dd, and split. These tricks could work because, AIUI,
tar and gzip do not require seeking within a file, and simply work with
the stream data. It might not work, if loopbacks are prevented from
working with a fifo- which apparently they are. I am not sure of the
solution at this time.

--
Douglas Mayne
Back to top
The Natural Philosopher
External


Since: Jul 09, 2004
Posts: 802



PostPosted: Thu Jul 12, 2007 8:57 pm    Post subject: Re: TAR & GZIP into multiple files [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

Christopher Ho wrote:
> Hi all,
>
> Is it possible to tar + gzip a huge directory and split into multiple
> archives ?
>
> The command I used to use is
>
> cd /data/ (which has a directory "output" under it)
>
> tar -czf /backup/data_May2007.tar.gz output
>
> This output folder contains about a million files and after
> compression the archive take the space about 15 Gb.
>
> I want to separate the archive into DVD size but I don't want to use
> up too much space to do all the operation.Any quick one line command I
> can achieve it? Thanks.
>


IIRC you can 'split' a tarfile using some command or other..yup.
'split' haha.

Probably best to do the tar and pipe it to split, then compress the
resultant files..not sure about reassembling a zipped file..guess it
SHOULD work..

However i'd be more inclined to split the data directory into subdirs,
and make a tar of each one..
Back to top
The Natural Philosopher
External


Since: Jul 09, 2004
Posts: 802



PostPosted: Thu Jul 12, 2007 9:30 pm    Post subject: Re: TAR & GZIP into multiple files [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

Douglas Mayne wrote:
> On Thu, 12 Jul 2007 15:06:53 +0000, Christopher Ho wrote:
>
>> Hi all,
>>
>> Is it possible to tar + gzip a huge directory and split into multiple
>> archives ?
>>
>> The command I used to use is
>>
>> cd /data/ (which has a directory "output" under it)
>>
>> tar -czf /backup/data_May2007.tar.gz output
>>
>> This output folder contains about a million files and after
>> compression the archive take the space about 15 Gb.
>>
>> I want to separate the archive into DVD size but I don't want to use
>> up too much space to do all the operation.Any quick one line command I
>> can achieve it? Thanks.
>>
> I am not sure about minimizing space used by potential intermediate files,
> but one trick is to use the command split. man split.
> Split breaks a larger file into blocks of the specified size.
>
> The individual block files can be written to DVD. The blocks must be
> pasted back together when restoring. Device mapper's "linear" module may
> be useful in virtually pasting the components back together. Without that
> trick, this is the basic idea:
>
> $ cat /mnt/dvd/part.xxx >>restore.tgz
>
> (For every block, and in sequence.)
>
> Notes:
> 1. The entire archive may be unusable if any one DVD fails or becomes
> unreadable.

This is a good reason to split the data before tarring..alphabetical
type wildcards can be used to eg. select files starting with a-g, h-o,
p-z etc..

>
> 2. Compute md5sums for each component written to DVD to provide a method
> in the future of ensuring that the disc is still readable.
>
> 3. Consider adding an encryption layer to system backups.
>
> 4. As far as minimizing space, there are probably some tricks that can be
> used to minimize intermediate files. These tricks would probably use
> fifo's, loopbacks, dd, and split. These tricks could work because, AIUI,
> tar and gzip do not require seeking within a file, and simply work with
> the stream data. It might not work, if loopbacks are prevented from
> working with a fifo- which apparently they are. I am not sure of the
> solution at this time.
>

At some level. whether its RAM paged out to swap, or system memory,
large parts of the backup process will use temporary disk.

Pipes certainly help..

Something like
cd /data
find . -iregex <regexp to select a subset> | tar cv- | gzip >
/backup./image1.gz
Back to top
ArameFarpado
External


Since: Apr 14, 2006
Posts: 195



PostPosted: Thu Jul 12, 2007 10:10 pm    Post subject: Re: TAR & GZIP into multiple files [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

Em Quinta, 12 de Julho de 2007 16:06, Christopher Ho escreveu:

> Hi all,
>
> Is it possible to tar + gzip a huge directory and split into multiple
> archives ?
>

rar is more apropriate for that job.
Back to top
Nico
External


Since: Jun 19, 2007
Posts: 29



PostPosted: Sat Jul 14, 2007 1:10 pm    Post subject: Re: TAR & GZIP into multiple files [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

On 12 Jul, 16:06, Christopher Ho wrote:
> Hi all,
>
> Is it possible to tar + gzip a huge directory and split into multiple
> archives ?
>
> The command I used to use is
>
> cd /data/ (which has a directory "output" under it)
>
> tar -czf /backup/data_May2007.tar.gz output

Think about creating a loop:

# cd /data
# for name in *; tar czvf /backup/data_May2007_$name.tar.gz $name;
done

You can get more complex than that to get any dotfiles in in /data,
but you get the idea.

You can also split the tar.gz into multiple pieces with "split", but I
don't recommend that. It means having to grab and recover all the
DVD's to recover things reliably.
Back to top
Display posts from previous:   
Post new topic   General Reply to Topic (not reply to a specific post)    Forums Home -> Setup All times are: Eastern Time (US & Canada)
Page 1 of 1

 
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum