diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..1ad7ca3 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,49 @@ +# Contributing to Transcriptase +We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's: + +- Reporting a bug +- Discussing the current state of the code +- Submitting a fix +- Proposing new features +- Becoming a maintainer + +## We Develop with Github +We use github to host code, to track issues and feature requests, as well as accept pull requests. + +## We Use [Github Flow](https://guides.github.com/introduction/flow/index.html), So All Code Changes Happen Through Pull Requests +Pull requests are the best way to propose changes to the codebase (we use [Github Flow](https://guides.github.com/introduction/flow/index.html)). We actively welcome your pull requests: + +1. Fork the repo and create your branch from `master`. +2. If you've added code that should be tested, add tests. +3. If you've changed APIs, update the documentation. +4. Ensure the test suite passes. +5. Make sure your code lints. +6. Issue that pull request! + +## Any contributions you make will be under the MIT Software License +In short, when you submit code changes, your submissions are understood to be under the same [MIT License](https://github.com/akamhy/waybackpy/blob/master/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern. + +## Report bugs using Github's [issues](https://github.com/akamhy/waybackpy/issues) +We use GitHub issues to track public bugs. Report a bug by [opening a new issue](https://github.com/akamhy/waybackpy/issues/new); it's that easy! + +## Write bug reports with detail, background, and sample code +**Great Bug Reports** tend to have: + +- A quick summary and/or background +- Steps to reproduce + - Be specific! + - Give sample code if you can. +- What you expected would happen +- What actually happens +- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work) + +People *love* thorough bug reports. I'm not even kidding. + +## Use a Consistent Coding Style +* You can try running `flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics` for style unification. + +## License +By contributing, you agree that your contributions will be licensed under its [MIT License](https://github.com/akamhy/waybackpy/blob/master/LICENSE). + +## References +This document is forked from [this gist](https://gist.github.com/briandk/3d2e8b3ec8daf5a27a62) by [briandk](https://github.com/briandk) which was itself adapted from the open-source contribution guidelines for [Facebook's Draft](https://github.com/facebook/draft-js/blob/a9316a723f9e918afde44dea68b5f9f39b7d9b00/CONTRIBUTING.md) diff --git a/README.md b/README.md index 323e09b..6f40907 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,25 @@ -# waybackpy +
+
+
-![contributions welcome](https://img.shields.io/static/v1.svg?label=Contributions&message=Welcome&color=0059b3&style=flat-square) +----------------- + +## Python package & CLI tool that interfaces with the Wayback Machine API. +[![pypi](https://img.shields.io/pypi/v/waybackpy.svg)](https://pypi.org/project/waybackpy/) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/akamhy/waybackpy/blob/master/LICENSE) [![Build Status](https://github.com/akamhy/waybackpy/workflows/CI/badge.svg)](https://github.com/akamhy/waybackpy/actions) [![codecov](https://codecov.io/gh/akamhy/waybackpy/branch/master/graph/badge.svg)](https://codecov.io/gh/akamhy/waybackpy) +[![contributions welcome](https://img.shields.io/static/v1.svg?label=Contributions&message=Welcome&color=0059b3&style=flat-square)](https://github.com/akamhy/waybackpy/blob/master/CONTRIBUTING.md) +[![Codacy Badge](https://api.codacy.com/project/badge/Grade/255459cede9341e39436ec8866d3fb65)](https://www.codacy.com/manual/akamhy/waybackpy?utm_source=github.com&utm_medium=referral&utm_content=akamhy/waybackpy&utm_campaign=Badge_Grade) [![Downloads](https://pepy.tech/badge/waybackpy/month)](https://pepy.tech/project/waybackpy) [![Release](https://img.shields.io/github/v/release/akamhy/waybackpy.svg)](https://github.com/akamhy/waybackpy/releases) -[![Codacy Badge](https://api.codacy.com/project/badge/Grade/255459cede9341e39436ec8866d3fb65)](https://www.codacy.com/manual/akamhy/waybackpy?utm_source=github.com&utm_medium=referral&utm_content=akamhy/waybackpy&utm_campaign=Badge_Grade) [![Maintainability](https://api.codeclimate.com/v1/badges/942f13d8177a56c1c906/maintainability)](https://codeclimate.com/github/akamhy/waybackpy/maintainability) [![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/) -[![pypi](https://img.shields.io/pypi/v/waybackpy.svg)](https://pypi.org/project/waybackpy/) -![PyPI - Python Version](https://img.shields.io/pypi/pyversions/waybackpy?style=flat-square) [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/akamhy/waybackpy/graphs/commit-activity) -![Repo size](https://img.shields.io/github/repo-size/akamhy/waybackpy.svg?label=Repo%20size&style=flat-square) -[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/akamhy/waybackpy/blob/master/LICENSE) +[![GitHub last commit](https://img.shields.io/github/last-commit/akamhy/waybackpy?color=blue&style=flat-square)](https://github.com/akamhy/waybackpy/commits/master) +![PyPI - Python Version](https://img.shields.io/pypi/pyversions/waybackpy?style=flat-square) -![Wayback Machine](https://raw.githubusercontent.com/akamhy/waybackpy/master/assets/waybackpy-colored%20284.png) -Waybackpy is a Python package that interfaces with [Internet Archive](https://en.wikipedia.org/wiki/Internet_Archive)'s [Wayback Machine](https://en.wikipedia.org/wiki/Wayback_Machine) API. Archive webpages and retrieve archived webpages easily. Table of contents ================= @@ -29,8 +32,8 @@ Table of contents * [Saving a webpage](#capturing-aka-saving-an-url-using-save) * [Retrieving archive](#retrieving-the-archive-for-an-url-using-archive_url) * [Retrieving the oldest archive](#retrieving-the-oldest-archive-for-an-url-using-oldest) - * [Retrieving the recent most/newest archive](#retrieving-the-newest-archive-for-an-url-using-newest) - * [Retrieving the JSON response of availability API](#retrieving-the-json-reponse-for-the-avaliblity-api-request) + * [Retrieving the latest/newest archive](#retrieving-the-newest-archive-for-an-url-using-newest) + * [Retrieving the JSON response of availability API](#retrieving-the-json-response-for-the-availability-api-request) * [Retrieving archive close to a specified year, month, day, hour, and minute](#retrieving-archive-close-to-a-specified-year-month-day-hour-and-minute-using-near) * [Get the content of webpage](#get-the-content-of-webpage-using-get) * [Count total archives for an URL](#count-total-archives-for-an-url-using-total_archives) @@ -49,8 +52,6 @@ Table of contents * [Tests](#tests) -* [Dependency](#dependency) - * [Packaging](#packaging) * [License](#license) @@ -75,7 +76,7 @@ pip install git+https://github.com/akamhy/waybackpy.git ### As a Python package -#### Capturing aka Saving an url using save() +#### Capturing aka Saving an URL using save() ```python import waybackpy @@ -151,7 +152,7 @@ https://web.archive.org/web/20201016150543/https://www.facebook.com/ Try this out in your browser @ -#### Retrieving the JSON reponse for the avaliblity API request +#### Retrieving the JSON response for the availability API request ```python import waybackpy @@ -219,7 +220,7 @@ print(github_archive_near_2018_4_july_9_2_am) https://web.archive.org/web/20180704090245/https://github.com/ ``` -The package doesn't support second argument yet. You are encourged to create a PR ;) +The package doesn't support the seconds' argument yet. You are encouraged to create a PR ;) Try this out in your browser @ @@ -373,10 +374,10 @@ https://web.archive.org/web/20120512142515/https://www.facebook.com/ #### Get the source code ```bash -waybackpy --url google.com --user_agent "my-unique-user-agent" --get url # Prints the source code of the url +waybackpy --url google.com --user_agent "my-unique-user-agent" --get url # Prints the source code of the URL waybackpy --url google.com --user_agent "my-unique-user-agent" --get oldest # Prints the source code of the oldest archive waybackpy --url google.com --user_agent "my-unique-user-agent" --get newest # Prints the source code of the newest archive -waybackpy --url google.com --user_agent "my-unique-user-agent" --get save # Save a new archive on wayback machine then print the source code of this archive. +waybackpy --url google.com --user_agent "my-unique-user-agent" --get save # Save a new archive on Wayback machine then print the source code of this archive. ``` Try this out in your browser @ @@ -402,7 +403,7 @@ waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --ali waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain -# Prints all known URLs under akamhy.github.io inclusing subdomain +# Prints all known URLs under akamhy.github.io including subdomain waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain --alive @@ -414,19 +415,29 @@ waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --sub ## Tests -[Here](https://github.com/akamhy/waybackpy/tree/master/tests) - To run tests locally: +1) Install or update the testing/coverage tools + ```bash -pip install -U pytest -pip install codecov -pip install pytest pytest-cov -cd tests -pytest --cov=../waybackpy -python -m codecov #For reporting coverage on Codecov +pip install codecov pytest pytest-cov -U ``` +2) Inside the repository run the following commands + +```bash +pytest --cov=waybackpy tests/ +``` + +3) To report coverage run + +```bash +bash <(curl -s https://codecov.io/bash) -t SECRET_CODECOV_TOKEN +``` + +You can find the tests [here](https://github.com/akamhy/waybackpy/tree/master/tests). + + ## Packaging 1. Increment version. diff --git a/assets/waybackpy colored.png b/assets/waybackpy colored.png deleted file mode 100644 index ea8f46a..0000000 Binary files a/assets/waybackpy colored.png and /dev/null differ diff --git a/assets/waybackpy-colored 284.png b/assets/waybackpy-colored 284.png deleted file mode 100644 index a256770..0000000 Binary files a/assets/waybackpy-colored 284.png and /dev/null differ diff --git a/assets/waybackpy_logo.svg b/assets/waybackpy_logo.svg new file mode 100644 index 0000000..1e989d8 --- /dev/null +++ b/assets/waybackpy_logo.svg @@ -0,0 +1,85 @@ + + + + + + + image/svg+xml + + + + + + + waybackpy + + + + + + diff --git a/index.rst b/index.rst deleted file mode 100644 index f6322bd..0000000 --- a/index.rst +++ /dev/null @@ -1,525 +0,0 @@ -waybackpy -========= - -|contributions welcome| |Build Status| |codecov| |Downloads| |Release| -|Codacy Badge| |Maintainability| |CodeFactor| |made-with-python| |pypi| -|PyPI - Python Version| |Maintenance| |Repo size| |License: MIT| - -.. figure:: https://raw.githubusercontent.com/akamhy/waybackpy/master/assets/waybackpy-colored%20284.png - :alt: Wayback Machine - - Wayback Machine -Waybackpy is a Python package that interfaces with `Internet -Archive `__'s `Wayback -Machine `__ API. Archive -webpages and retrieve archived webpages easily. - -Table of contents -================= - -.. raw:: html - - - -- `Installation <#installation>`__ - -- `Usage <#usage>`__ -- `As a Python package <#as-a-python-package>`__ - - - `Saving a webpage <#capturing-aka-saving-an-url-using-save>`__ - - `Retrieving - archive <#retrieving-the-archive-for-an-url-using-archive_url>`__ - - `Retrieving the oldest - archive <#retrieving-the-oldest-archive-for-an-url-using-oldest>`__ - - `Retrieving the recent most/newest - archive <#retrieving-the-newest-archive-for-an-url-using-newest>`__ - - `Retrieving the JSON response of availability - API <#retrieving-the-json-reponse-for-the-avaliblity-api-request>`__ - - `Retrieving archive close to a specified year, month, day, hour, - and - minute <#retrieving-archive-close-to-a-specified-year-month-day-hour-and-minute-using-near>`__ - - `Get the content of - webpage <#get-the-content-of-webpage-using-get>`__ - - `Count total archives for an - URL <#count-total-archives-for-an-url-using-total_archives>`__ - - `List of URLs that Wayback Machine knows and has archived for a - domain - name <#list-of-urls-that-wayback-machine-knows-and-has-archived-for-a-domain-name>`__ - -- `With the Command-line - interface <#with-the-command-line-interface>`__ - - - `Saving webpage <#save>`__ - - `Archive URL <#get-archive-url>`__ - - `Oldest archive URL <#oldest-archive>`__ - - `Newest archive URL <#newest-archive>`__ - - `JSON response of API <#get-json-data-of-avaialblity-api>`__ - - `Total archives <#total-number-of-archives>`__ - - `Archive near specified time <#archive-near-time>`__ - - `Get the source code <#get-the-source-code>`__ - - `Fetch all the URLs that the Wayback Machine knows for a - domain <#fetch-all-the-urls-that-the-wayback-machine-knows-for-a-domain>`__ - -- `Tests <#tests>`__ - -- `Dependency <#dependency>`__ - -- `Packaging <#packaging>`__ - -- `License <#license>`__ - -.. raw:: html - - - -Installation ------------- - -Using `pip `__: - -.. code:: bash - - pip install waybackpy - -or direct from this repository using git. - -.. code:: bash - - pip install git+https://github.com/akamhy/waybackpy.git - -Usage ------ - -As a Python package -~~~~~~~~~~~~~~~~~~~ - -Capturing aka Saving an url using save() -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: python - - import waybackpy - - url = "https://en.wikipedia.org/wiki/Multivariable_calculus" - user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0" - - waybackpy_url_obj = waybackpy.Url(url, user_agent) - archive = waybackpy_url_obj.save() - print(archive) - -.. code:: bash - - https://web.archive.org/web/20201016171808/https://en.wikipedia.org/wiki/Multivariable_calculus - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPySaveExample\ - -Retrieving the archive for an URL using archive\_url -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: python - - import waybackpy - - url = "https://www.google.com/" - user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:40.0) Gecko/20100101 Firefox/40.0" - - waybackpy_url_obj = waybackpy.Url(url, user_agent) - archive_url = waybackpy_url_obj.archive_url - print(archive_url) - -.. code:: bash - - https://web.archive.org/web/20201016153320/https://www.google.com/ - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyArchiveUrl\ - -Retrieving the oldest archive for an URL using oldest() -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: python - - import waybackpy - - url = "https://www.google.com/" - user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:40.0) Gecko/20100101 Firefox/40.0" - - waybackpy_url_obj = waybackpy.Url(url, user_agent) - oldest_archive_url = waybackpy_url_obj.oldest() - print(oldest_archive_url) - -.. code:: bash - - http://web.archive.org/web/19981111184551/http://google.com:80/ - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyOldestExample\ - -Retrieving the newest archive for an URL using newest() -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: python - - import waybackpy - - url = "https://www.facebook.com/" - user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0" - - waybackpy_url_obj = waybackpy.Url(url, user_agent) - newest_archive_url = waybackpy_url_obj.newest() - print(newest_archive_url) - -.. code:: bash - - https://web.archive.org/web/20201016150543/https://www.facebook.com/ - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyNewestExample\ - -Retrieving the JSON reponse for the avaliblity API request -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: python - - import waybackpy - - url = "https://www.facebook.com/" - user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0" - - waybackpy_url_obj = waybackpy.Url(url, user_agent) - json_dict = waybackpy_url_obj.JSON - print(json_dict) - -.. code:: javascript - - {'url': 'https://www.facebook.com/', 'archived_snapshots': {'closest': {'available': True, 'url': 'http://web.archive.org/web/20201016150543/https://www.facebook.com/', 'timestamp': '20201016150543', 'status': '200'}}} - -Try this out in your browser @ https://repl.it/@akamhy/WaybackPyJSON\ - -Retrieving archive close to a specified year, month, day, hour, and minute using near() -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: python - - from waybackpy import Url - - user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Firefox/38.0" - url = "https://github.com/" - - waybackpy_url_obj = Url(url, user_agent) - - # Do not pad (don't use zeros in the month, year, day, minute, and hour arguments). e.g. For January, set month = 1 and not month = 01. - -.. code:: python - - github_archive_near_2010 = waybackpy_url_obj.near(year=2010) - print(github_archive_near_2010) - -.. code:: bash - - https://web.archive.org/web/20101018053604/http://github.com:80/ - -.. code:: python - - github_archive_near_2011_may = waybackpy_url_obj.near(year=2011, month=5) - print(github_archive_near_2011_may) - -.. code:: bash - - https://web.archive.org/web/20110518233639/https://github.com/ - -.. code:: python - - github_archive_near_2015_january_26 = waybackpy_url_obj.near(year=2015, month=1, day=26) - print(github_archive_near_2015_january_26) - -.. code:: bash - - https://web.archive.org/web/20150125102636/https://github.com/ - -.. code:: python - - github_archive_near_2018_4_july_9_2_am = waybackpy_url_obj.near(year=2018, month=7, day=4, hour=9, minute=2) - print(github_archive_near_2018_4_july_9_2_am) - -.. code:: bash - - https://web.archive.org/web/20180704090245/https://github.com/ - -The package doesn't support second argument yet. You are encourged to -create a PR ;) - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyNearExample\ - -Get the content of webpage using get() -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: python - - import waybackpy - - google_url = "https://www.google.com/" - - User_Agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36" - - waybackpy_url_object = waybackpy.Url(google_url, User_Agent) - - - # If no argument is passed in get(), it gets the source of the Url used to create the object. - current_google_url_source = waybackpy_url_object.get() - print(current_google_url_source) - - - # The following chunk of code will force a new archive of google.com and get the source of the archived page. - # waybackpy_url_object.save() type is string. - google_newest_archive_source = waybackpy_url_object.get(waybackpy_url_object.save()) - print(google_newest_archive_source) - - - # waybackpy_url_object.oldest() type is str, it's oldest archive of google.com - google_oldest_archive_source = waybackpy_url_object.get(waybackpy_url_object.oldest()) - print(google_oldest_archive_source) - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyGetExample#main.py\ - -Count total archives for an URL using total\_archives() -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: python - - import waybackpy - - URL = "https://en.wikipedia.org/wiki/Python (programming language)" - UA = "Mozilla/5.0 (iPad; CPU OS 8_1_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B435 Safari/600.1.4" - - waybackpy_url_object = waybackpy.Url(url=URL, user_agent=UA) - - archive_count = waybackpy_url_object.total_archives() - - print(archive_count) # total_archives() returns an int - -.. code:: bash - - 2516 - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyTotalArchivesExample\ - -List of URLs that Wayback Machine knows and has archived for a domain name -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -1) If alive=True is set, waybackpy will check all URLs to identify the - alive URLs. Don't use with popular websites like google or it would - take too long. -2) To include URLs from subdomain set sundomain=True - -.. code:: python - - import waybackpy - - URL = "akamhy.github.io" - UA = "Mozilla/5.0 (iPad; CPU OS 8_1_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B435 Safari/600.1.4" - - waybackpy_url_object = waybackpy.Url(url=URL, user_agent=UA) - known_urls = waybackpy_url_object.known_urls(alive=True, subdomain=False) # alive and subdomain are optional. - print(known_urls) # known_urls() returns list of URLs - -.. code:: bash - - ['http://akamhy.github.io', - 'https://akamhy.github.io/waybackpy/', - 'https://akamhy.github.io/waybackpy/assets/css/style.css?v=a418a4e4641a1dbaad8f3bfbf293fad21a75ff11', - 'https://akamhy.github.io/waybackpy/assets/css/style.css?v=f881705d00bf47b5bf0c58808efe29eecba2226c'] - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyKnownURLsToWayBackMachineExample#main.py\ - -With the Command-line interface -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Save -^^^^ - -.. code:: bash - - $ waybackpy --url "https://en.wikipedia.org/wiki/Social_media" --user_agent "my-unique-user-agent" --save - https://web.archive.org/web/20200719062108/https://en.wikipedia.org/wiki/Social_media - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyBashSave\ - -Get archive URL -^^^^^^^^^^^^^^^ - -.. code:: bash - - $ waybackpy --url "https://en.wikipedia.org/wiki/SpaceX" --user_agent "my-unique-user-agent" --archive_url - https://web.archive.org/web/20201007132458/https://en.wikipedia.org/wiki/SpaceX - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyBashArchiveUrl\ - -Oldest archive -^^^^^^^^^^^^^^ - -.. code:: bash - - $ waybackpy --url "https://en.wikipedia.org/wiki/SpaceX" --user_agent "my-unique-user-agent" --oldest - https://web.archive.org/web/20040803000845/http://en.wikipedia.org:80/wiki/SpaceX - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyBashOldest\ - -Newest archive -^^^^^^^^^^^^^^ - -.. code:: bash - - $ waybackpy --url "https://en.wikipedia.org/wiki/YouTube" --user_agent "my-unique-user-agent" --newest - https://web.archive.org/web/20200606044708/https://en.wikipedia.org/wiki/YouTube - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyBashNewest\ - -Get JSON data of avaialblity API -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: bash - - waybackpy --url "https://en.wikipedia.org/wiki/SpaceX" --user_agent "my-unique-user-agent" --json - -.. code:: javascript - - {'archived_snapshots': {'closest': {'timestamp': '20201007132458', 'status': '200', 'available': True, 'url': 'http://web.archive.org/web/20201007132458/https://en.wikipedia.org/wiki/SpaceX'}}, 'url': 'https://en.wikipedia.org/wiki/SpaceX'} - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyBashJSON\ - -Total number of archives -^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: bash - - $ waybackpy --url "https://en.wikipedia.org/wiki/Linux_kernel" --user_agent "my-unique-user-agent" --total - 853 - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyBashTotal\ - -Archive near time -^^^^^^^^^^^^^^^^^ - -.. code:: bash - - $ waybackpy --url facebook.com --user_agent "my-unique-user-agent" --near --year 2012 --month 5 --day 12 - https://web.archive.org/web/20120512142515/https://www.facebook.com/ - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyBashNear\ - -Get the source code -^^^^^^^^^^^^^^^^^^^ - -.. code:: bash - - waybackpy --url google.com --user_agent "my-unique-user-agent" --get url # Prints the source code of the url - waybackpy --url google.com --user_agent "my-unique-user-agent" --get oldest # Prints the source code of the oldest archive - waybackpy --url google.com --user_agent "my-unique-user-agent" --get newest # Prints the source code of the newest archive - waybackpy --url google.com --user_agent "my-unique-user-agent" --get save # Save a new archive on wayback machine then print the source code of this archive. - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackPyBashGet\ - -Fetch all the URLs that the Wayback Machine knows for a domain -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -1) You can add the '--alive' flag to only fetch alive links. -2) You can add the '--subdomain' flag to add subdomains. -3) '--alive' and '--subdomain' flags can be used simultaneously. -4) All links will be saved in a file, and the file will be created in - the current working directory. - -.. code:: bash - - pip install waybackpy - - # Ignore the above installation line. - - waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls - # Prints all known URLs under akamhy.github.io - - - waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --alive - # Prints all known URLs under akamhy.github.io which are still working and not dead links. - - - waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain - # Prints all known URLs under akamhy.github.io inclusing subdomain - - - waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain --alive - # Prints all known URLs under akamhy.github.io including subdomain which are not dead links and still alive. - -Try this out in your browser @ -https://repl.it/@akamhy/WaybackpyKnownUrlsFromWaybackMachine#main.sh\ - -Tests ------ - -`Here `__ - -To run tests locally: - -.. code:: bash - - pip install -U pytest - pip install codecov - pip install pytest pytest-cov - cd tests - pytest --cov=../waybackpy - python -m codecov #For reporting coverage on Codecov - -Packaging ---------- - -1. Increment version. - -2. Build package ``python setup.py sdist bdist_wheel``. - -3. Sign & upload the package ``twine upload -s dist/*``. - -License -------- - -Released under the MIT License. See -`license `__ -for details. - -.. |contributions welcome| image:: https://img.shields.io/static/v1.svg?label=Contributions&message=Welcome&color=0059b3&style=flat-square -.. |Build Status| image:: https://img.shields.io/travis/akamhy/waybackpy.svg?label=Travis%20CI&logo=travis&style=flat-square - :target: https://travis-ci.org/akamhy/waybackpy -.. |codecov| image:: https://codecov.io/gh/akamhy/waybackpy/branch/master/graph/badge.svg - :target: https://codecov.io/gh/akamhy/waybackpy -.. |Downloads| image:: https://pepy.tech/badge/waybackpy/month - :target: https://pepy.tech/project/waybackpy/month -.. |Release| image:: https://img.shields.io/github/v/release/akamhy/waybackpy.svg - :target: https://github.com/akamhy/waybackpy/releases -.. |Codacy Badge| image:: https://api.codacy.com/project/badge/Grade/255459cede9341e39436ec8866d3fb65 - :target: https://www.codacy.com/manual/akamhy/waybackpy?utm_source=github.com&utm_medium=referral&utm_content=akamhy/waybackpy&utm_campaign=Badge_Grade -.. |Maintainability| image:: https://api.codeclimate.com/v1/badges/942f13d8177a56c1c906/maintainability - :target: https://codeclimate.com/github/akamhy/waybackpy/maintainability -.. |CodeFactor| image:: https://www.codefactor.io/repository/github/akamhy/waybackpy/badge - :target: https://www.codefactor.io/repository/github/akamhy/waybackpy -.. |made-with-python| image:: https://img.shields.io/badge/Made%20with-Python-1f425f.svg - :target: https://www.python.org/ -.. |pypi| image:: https://img.shields.io/pypi/v/waybackpy.svg - :target: https://pypi.org/project/waybackpy/ -.. |PyPI - Python Version| image:: https://img.shields.io/pypi/pyversions/waybackpy?style=flat-square -.. |Maintenance| image:: https://img.shields.io/badge/Maintained%3F-yes-green.svg - :target: https://github.com/akamhy/waybackpy/graphs/commit-activity -.. |Repo size| image:: https://img.shields.io/github/repo-size/akamhy/waybackpy.svg?label=Repo%20size&style=flat-square -.. |License: MIT| image:: https://img.shields.io/badge/License-MIT-yellow.svg - :target: https://github.com/akamhy/waybackpy/blob/master/LICENSE diff --git a/setup.py b/setup.py index 3e4491c..a1f62f6 100644 --- a/setup.py +++ b/setup.py @@ -45,7 +45,8 @@ setup( ] }, project_urls={ - 'Documentation': 'https://waybackpy.readthedocs.io', + 'Documentation': 'https://akamhy.github.io/waybackpy/', 'Source': 'https://github.com/akamhy/waybackpy', + 'Tracker': 'https://github.com/akamhy/waybackpy/issues', }, )