Benjamin_Loison/waybackpy
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity
Fork of https://github.com/akamhy/waybackpy Wayback Machine API interface & a command-line tool
333 Commits 1 Branch 37 Tags 1.1 MiB
Python 100%
d549d31421
Go to file
Akash Mahanty d549d31421 improve save method, now we know that 302 errors indicates that wayback machine is archiving the URL and hasn't yet archived. We construct an artifical archive with the current UTC time and check for HTTP status code 20* or 30*. If we verify the archival, we return the artifical archive. The artificial archive will automatically point to the new archive or in best case will be the new archive after some time.
2021-01-16 10:47:43 +05:30
.github/workflows remove print stmnt, update ci 2020-12-13 16:37:35 +05:30
assets mimify the logo, remove ugly old logos 2021-01-15 18:14:48 +05:30
tests better error messages(str), check latest version before asking for an upgrade and rm alive checking 2021-01-15 16:47:26 +05:30
waybackpy improve save method, now we know that 302 errors indicates that wayback machine is archiving the URL and hasn't yet archived. We construct an artifical archive with the current UTC time and check for HTTP status code 20* or 30*. If we verify the archival, we return the artifical archive. The artificial archive will automatically point to the new archive or in best case will be the new archive after some time. 2021-01-16 10:47:43 +05:30
_config.yml now using requests lib as it handles errors nicely 2020-12-13 15:05:57 +05:30
.gitignore removed JSON from init, this was resulting in too much unnecessary taffic. Some users who are thousands of URLs were blocked by IA (#53) 2021-01-01 16:38:57 +05:30
.pep8speaks.yml Create .pep8speaks.yml 2020-12-13 17:58:09 +05:30
.pyup.yml create pyup.io config file (#44) 2020-12-13 22:31:49 +05:30
.whitesource Add .whitesource configuration file (#6) 2020-05-05 09:33:50 +05:30
CONTRIBUTING.md Update CONTRIBUTING.md 2020-12-14 00:16:29 +05:30
CONTRIBUTORS.md + dequeued0 2021-01-12 10:52:41 +05:30
LICENSE + https://github.com/akamhy/waybackpy/graphs/contributors 2020-11-04 08:09:30 +05:30
README.md + Supported Features 2021-01-11 23:01:18 +05:30
requirements.txt now using requests lib as it handles errors nicely 2020-12-13 15:05:57 +05:30
setup.cfg Code style improvements (#20) 2020-07-22 10:09:14 +05:30
setup.py v2.4.1 2021-01-12 10:18:09 +05:30

README.md


Python package & CLI tool that interfaces with the Wayback Machine API

pypi Build Status Codacy Badge codecov Contributions Welcome Downloads GitHub lastest commit PyPI - Python Version

Installation

Using pip:

pip install waybackpy

Install directly from GitHub:

pip install git+https://github.com/akamhy/waybackpy.git

Supported Features

  • Archive webpage
  • Retrieve all archives of a webpage/domain
  • Retrieve archive close to a date or timestamp
  • Retrieve all archives which have a particular prefix
  • Get source code of the archive easily
  • CDX API support

Usage

As a Python package

>>> import waybackpy

>>> url = "https://en.wikipedia.org/wiki/Multivariable_calculus"
>>> user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"

>>> wayback = waybackpy.Url(url, user_agent)

>>> archive = wayback.save()
>>> archive.archive_url
'https://web.archive.org/web/20210104173410/https://en.wikipedia.org/wiki/Multivariable_calculus'

>>> archive.timestamp
datetime.datetime(2021, 1, 4, 17, 35, 12, 691741)

>>> oldest_archive = wayback.oldest()
>>> oldest_archive.archive_url
'https://web.archive.org/web/20050422130129/http://en.wikipedia.org:80/wiki/Multivariable_calculus'

>>> archive_close_to_2010_feb = wayback.near(year=2010, month=2)
>>> archive_close_to_2010_feb.archive_url
'https://web.archive.org/web/20100215001541/http://en.wikipedia.org:80/wiki/Multivariable_calculus'

>>> wayback.newest().archive_url
'https://web.archive.org/web/20210104173410/https://en.wikipedia.org/wiki/Multivariable_calculus'

Full Python package documentation can be found at https://github.com/akamhy/waybackpy/wiki/Python-package-docs.

As a CLI tool

$ waybackpy --save --url "https://en.wikipedia.org/wiki/Social_media" --user_agent "my-unique-user-agent"
https://web.archive.org/web/20200719062108/https://en.wikipedia.org/wiki/Social_media

$ waybackpy --oldest --url "https://en.wikipedia.org/wiki/Humanoid" --user_agent "my-unique-user-agent"
https://web.archive.org/web/20040415020811/http://en.wikipedia.org:80/wiki/Humanoid

$ waybackpy --newest --url "https://en.wikipedia.org/wiki/Remote_sensing" --user_agent "my-unique-user-agent"
https://web.archive.org/web/20201221130522/https://en.wikipedia.org/wiki/Remote_sensing

$ waybackpy --total --url "https://en.wikipedia.org/wiki/Linux_kernel" --user_agent "my-unique-user-agent"
1904

$ waybackpy --known_urls --url akamhy.github.io --user_agent "my-unique-user-agent"
https://akamhy.github.io
https://akamhy.github.io/assets/js/scale.fix.js
https://akamhy.github.io/favicon.ico
https://akamhy.github.io/robots.txt
https://akamhy.github.io/waybackpy/

'akamhy.github.io-10-urls-m2a24y.txt' saved in current working directory

Full CLI documentation can be found at https://github.com/akamhy/waybackpy/wiki/CLI-docs.

License

License: MIT

Released under the MIT License. See license for details.