If you want to have an uncompressed WARC file, use the -no-warc-compression option: This is a gzipped WARC file that contains the request and response headers (of the initial redirect and of the Wiki homepage) and the html data. This will download the file to index.html, but it will also create a file. To download a file and save the request and response data to a WARC file, run this: and (b) wget overwrites WARC files (but not idx files) if you're not very careful. In particular: (a) only HTTP(S) requests and replies are stored not auxiliary content such as DNS queries, PKI used to negotiate HTTPS connections and etc. Wget's WARC file support (as at 1.19.5 / RHEL8) is relatively incomplete and immature compared to other specialist archiving systems.
There is no need to remove these headers afterwards to produce a clean copy: the mirror produced by Wget is usable without post-processing. There is an additional advantage: if Wget writes these headers to a WARC file, it is no longer necessary to use the -save-headers to save them at the top of each downloaded file. It also provides a clean way to store redirects and 404 responses. With the WARC format, both the request and the response headers get saved.
#Wget output directory archive#
Since version 1.14 Wget supports writing to a WARC file (Web ARChive file format) file, just like Heritrix and other archiving tools. You also lose the response headers that don't produce an HTML page: Wget doesn't save redirects and 404 responses. With a few tricks you can keep the response headers, but there is no option to save the request headers. From the discussion about Working with ARCHIVE.ORG, we learn that it is important to save not just files but also HTTP headers.
0 kommentar(er)
