Improve the appearance of readme (#45)
* replaced text header wth image * svg * Update README.md * Update README.md * Update README.md * level 2 * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Create CONTRIBUTING.md * Update README.md * Add files via upload * Update README.md * Delete waybackpy-colored 284.png * Delete waybackpy colored.png * Update README.md * Update index.rst * Update index.rst * Update index.rst * Update setup.py * Delete index.rst * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md
This commit is contained in:
parent
8125526061
commit
3c53b411b0
49
CONTRIBUTING.md
Normal file
49
CONTRIBUTING.md
Normal file
@ -0,0 +1,49 @@
|
||||
# Contributing to Transcriptase
|
||||
We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:
|
||||
|
||||
- Reporting a bug
|
||||
- Discussing the current state of the code
|
||||
- Submitting a fix
|
||||
- Proposing new features
|
||||
- Becoming a maintainer
|
||||
|
||||
## We Develop with Github
|
||||
We use github to host code, to track issues and feature requests, as well as accept pull requests.
|
||||
|
||||
## We Use [Github Flow](https://guides.github.com/introduction/flow/index.html), So All Code Changes Happen Through Pull Requests
|
||||
Pull requests are the best way to propose changes to the codebase (we use [Github Flow](https://guides.github.com/introduction/flow/index.html)). We actively welcome your pull requests:
|
||||
|
||||
1. Fork the repo and create your branch from `master`.
|
||||
2. If you've added code that should be tested, add tests.
|
||||
3. If you've changed APIs, update the documentation.
|
||||
4. Ensure the test suite passes.
|
||||
5. Make sure your code lints.
|
||||
6. Issue that pull request!
|
||||
|
||||
## Any contributions you make will be under the MIT Software License
|
||||
In short, when you submit code changes, your submissions are understood to be under the same [MIT License](https://github.com/akamhy/waybackpy/blob/master/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern.
|
||||
|
||||
## Report bugs using Github's [issues](https://github.com/akamhy/waybackpy/issues)
|
||||
We use GitHub issues to track public bugs. Report a bug by [opening a new issue](https://github.com/akamhy/waybackpy/issues/new); it's that easy!
|
||||
|
||||
## Write bug reports with detail, background, and sample code
|
||||
**Great Bug Reports** tend to have:
|
||||
|
||||
- A quick summary and/or background
|
||||
- Steps to reproduce
|
||||
- Be specific!
|
||||
- Give sample code if you can.
|
||||
- What you expected would happen
|
||||
- What actually happens
|
||||
- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)
|
||||
|
||||
People *love* thorough bug reports. I'm not even kidding.
|
||||
|
||||
## Use a Consistent Coding Style
|
||||
* You can try running `flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics` for style unification.
|
||||
|
||||
## License
|
||||
By contributing, you agree that your contributions will be licensed under its [MIT License](https://github.com/akamhy/waybackpy/blob/master/LICENSE).
|
||||
|
||||
## References
|
||||
This document is forked from [this gist](https://gist.github.com/briandk/3d2e8b3ec8daf5a27a62) by [briandk](https://github.com/briandk) which was itself adapted from the open-source contribution guidelines for [Facebook's Draft](https://github.com/facebook/draft-js/blob/a9316a723f9e918afde44dea68b5f9f39b7d9b00/CONTRIBUTING.md)
|
65
README.md
65
README.md
@ -1,22 +1,25 @@
|
||||
# waybackpy
|
||||
<div align="center">
|
||||
<img src="https://raw.githubusercontent.com/akamhy/waybackpy/improve-appearance-of-readme/assets/waybackpy_logo.svg"><br>
|
||||
</div>
|
||||
|
||||
![contributions welcome](https://img.shields.io/static/v1.svg?label=Contributions&message=Welcome&color=0059b3&style=flat-square)
|
||||
-----------------
|
||||
|
||||
## Python package & CLI tool that interfaces with the Wayback Machine API.
|
||||
[![pypi](https://img.shields.io/pypi/v/waybackpy.svg)](https://pypi.org/project/waybackpy/)
|
||||
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/akamhy/waybackpy/blob/master/LICENSE)
|
||||
[![Build Status](https://github.com/akamhy/waybackpy/workflows/CI/badge.svg)](https://github.com/akamhy/waybackpy/actions)
|
||||
[![codecov](https://codecov.io/gh/akamhy/waybackpy/branch/master/graph/badge.svg)](https://codecov.io/gh/akamhy/waybackpy)
|
||||
[![contributions welcome](https://img.shields.io/static/v1.svg?label=Contributions&message=Welcome&color=0059b3&style=flat-square)](https://github.com/akamhy/waybackpy/blob/master/CONTRIBUTING.md)
|
||||
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/255459cede9341e39436ec8866d3fb65)](https://www.codacy.com/manual/akamhy/waybackpy?utm_source=github.com&utm_medium=referral&utm_content=akamhy/waybackpy&utm_campaign=Badge_Grade)
|
||||
[![Downloads](https://pepy.tech/badge/waybackpy/month)](https://pepy.tech/project/waybackpy)
|
||||
[![Release](https://img.shields.io/github/v/release/akamhy/waybackpy.svg)](https://github.com/akamhy/waybackpy/releases)
|
||||
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/255459cede9341e39436ec8866d3fb65)](https://www.codacy.com/manual/akamhy/waybackpy?utm_source=github.com&utm_medium=referral&utm_content=akamhy/waybackpy&utm_campaign=Badge_Grade)
|
||||
[![Maintainability](https://api.codeclimate.com/v1/badges/942f13d8177a56c1c906/maintainability)](https://codeclimate.com/github/akamhy/waybackpy/maintainability)
|
||||
[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)
|
||||
[![pypi](https://img.shields.io/pypi/v/waybackpy.svg)](https://pypi.org/project/waybackpy/)
|
||||
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/waybackpy?style=flat-square)
|
||||
[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/akamhy/waybackpy/graphs/commit-activity)
|
||||
![Repo size](https://img.shields.io/github/repo-size/akamhy/waybackpy.svg?label=Repo%20size&style=flat-square)
|
||||
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/akamhy/waybackpy/blob/master/LICENSE)
|
||||
[![GitHub last commit](https://img.shields.io/github/last-commit/akamhy/waybackpy?color=blue&style=flat-square)](https://github.com/akamhy/waybackpy/commits/master)
|
||||
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/waybackpy?style=flat-square)
|
||||
|
||||
![Wayback Machine](https://raw.githubusercontent.com/akamhy/waybackpy/master/assets/waybackpy-colored%20284.png)
|
||||
|
||||
Waybackpy is a Python package that interfaces with [Internet Archive](https://en.wikipedia.org/wiki/Internet_Archive)'s [Wayback Machine](https://en.wikipedia.org/wiki/Wayback_Machine) API. Archive webpages and retrieve archived webpages easily.
|
||||
|
||||
Table of contents
|
||||
=================
|
||||
@ -29,8 +32,8 @@ Table of contents
|
||||
* [Saving a webpage](#capturing-aka-saving-an-url-using-save)
|
||||
* [Retrieving archive](#retrieving-the-archive-for-an-url-using-archive_url)
|
||||
* [Retrieving the oldest archive](#retrieving-the-oldest-archive-for-an-url-using-oldest)
|
||||
* [Retrieving the recent most/newest archive](#retrieving-the-newest-archive-for-an-url-using-newest)
|
||||
* [Retrieving the JSON response of availability API](#retrieving-the-json-reponse-for-the-avaliblity-api-request)
|
||||
* [Retrieving the latest/newest archive](#retrieving-the-newest-archive-for-an-url-using-newest)
|
||||
* [Retrieving the JSON response of availability API](#retrieving-the-json-response-for-the-availability-api-request)
|
||||
* [Retrieving archive close to a specified year, month, day, hour, and minute](#retrieving-archive-close-to-a-specified-year-month-day-hour-and-minute-using-near)
|
||||
* [Get the content of webpage](#get-the-content-of-webpage-using-get)
|
||||
* [Count total archives for an URL](#count-total-archives-for-an-url-using-total_archives)
|
||||
@ -49,8 +52,6 @@ Table of contents
|
||||
|
||||
* [Tests](#tests)
|
||||
|
||||
* [Dependency](#dependency)
|
||||
|
||||
* [Packaging](#packaging)
|
||||
|
||||
* [License](#license)
|
||||
@ -75,7 +76,7 @@ pip install git+https://github.com/akamhy/waybackpy.git
|
||||
|
||||
### As a Python package
|
||||
|
||||
#### Capturing aka Saving an url using save()
|
||||
#### Capturing aka Saving an URL using save()
|
||||
|
||||
```python
|
||||
import waybackpy
|
||||
@ -151,7 +152,7 @@ https://web.archive.org/web/20201016150543/https://www.facebook.com/
|
||||
|
||||
<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyNewestExample></sub>
|
||||
|
||||
#### Retrieving the JSON reponse for the avaliblity API request
|
||||
#### Retrieving the JSON response for the availability API request
|
||||
|
||||
```python
|
||||
import waybackpy
|
||||
@ -219,7 +220,7 @@ print(github_archive_near_2018_4_july_9_2_am)
|
||||
https://web.archive.org/web/20180704090245/https://github.com/
|
||||
```
|
||||
|
||||
<sub>The package doesn't support second argument yet. You are encourged to create a PR ;)</sub>
|
||||
<sub>The package doesn't support the seconds' argument yet. You are encouraged to create a PR ;)</sub>
|
||||
|
||||
<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyNearExample></sub>
|
||||
|
||||
@ -373,10 +374,10 @@ https://web.archive.org/web/20120512142515/https://www.facebook.com/
|
||||
#### Get the source code
|
||||
|
||||
```bash
|
||||
waybackpy --url google.com --user_agent "my-unique-user-agent" --get url # Prints the source code of the url
|
||||
waybackpy --url google.com --user_agent "my-unique-user-agent" --get url # Prints the source code of the URL
|
||||
waybackpy --url google.com --user_agent "my-unique-user-agent" --get oldest # Prints the source code of the oldest archive
|
||||
waybackpy --url google.com --user_agent "my-unique-user-agent" --get newest # Prints the source code of the newest archive
|
||||
waybackpy --url google.com --user_agent "my-unique-user-agent" --get save # Save a new archive on wayback machine then print the source code of this archive.
|
||||
waybackpy --url google.com --user_agent "my-unique-user-agent" --get save # Save a new archive on Wayback machine then print the source code of this archive.
|
||||
```
|
||||
|
||||
<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyBashGet></sub>
|
||||
@ -402,7 +403,7 @@ waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --ali
|
||||
|
||||
|
||||
waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain
|
||||
# Prints all known URLs under akamhy.github.io inclusing subdomain
|
||||
# Prints all known URLs under akamhy.github.io including subdomain
|
||||
|
||||
|
||||
waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain --alive
|
||||
@ -414,19 +415,29 @@ waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --sub
|
||||
|
||||
## Tests
|
||||
|
||||
[Here](https://github.com/akamhy/waybackpy/tree/master/tests)
|
||||
|
||||
To run tests locally:
|
||||
|
||||
1) Install or update the testing/coverage tools
|
||||
|
||||
```bash
|
||||
pip install -U pytest
|
||||
pip install codecov
|
||||
pip install pytest pytest-cov
|
||||
cd tests
|
||||
pytest --cov=../waybackpy
|
||||
python -m codecov #For reporting coverage on Codecov
|
||||
pip install codecov pytest pytest-cov -U
|
||||
```
|
||||
|
||||
2) Inside the repository run the following commands
|
||||
|
||||
```bash
|
||||
pytest --cov=waybackpy tests/
|
||||
```
|
||||
|
||||
3) To report coverage run
|
||||
|
||||
```bash
|
||||
bash <(curl -s https://codecov.io/bash) -t SECRET_CODECOV_TOKEN
|
||||
```
|
||||
|
||||
You can find the tests [here](https://github.com/akamhy/waybackpy/tree/master/tests).
|
||||
|
||||
|
||||
## Packaging
|
||||
|
||||
1. Increment version.
|
||||
|
Binary file not shown.
Before Width: | Height: | Size: 56 KiB |
Binary file not shown.
Before Width: | Height: | Size: 18 KiB |
85
assets/waybackpy_logo.svg
Normal file
85
assets/waybackpy_logo.svg
Normal file
@ -0,0 +1,85 @@
|
||||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<svg
|
||||
xmlns:dc="http://purl.org/dc/elements/1.1/"
|
||||
xmlns:cc="http://creativecommons.org/ns#"
|
||||
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
|
||||
xmlns:svg="http://www.w3.org/2000/svg"
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
id="svg8"
|
||||
version="1.1"
|
||||
viewBox="0 0 176.61171 41.907883"
|
||||
height="41.907883mm"
|
||||
width="176.61171mm">
|
||||
<defs
|
||||
id="defs2" />
|
||||
<metadata
|
||||
id="metadata5">
|
||||
<rdf:RDF>
|
||||
<cc:Work
|
||||
rdf:about="">
|
||||
<dc:format>image/svg+xml</dc:format>
|
||||
<dc:type
|
||||
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
|
||||
<dc:title></dc:title>
|
||||
</cc:Work>
|
||||
</rdf:RDF>
|
||||
</metadata>
|
||||
<g
|
||||
transform="translate(-0.74835286,-98.31182)"
|
||||
id="layer1">
|
||||
<flowRoot
|
||||
transform="scale(0.26458333)"
|
||||
style="font-style:normal;font-weight:normal;font-size:40px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none"
|
||||
id="flowRoot4598"
|
||||
xml:space="preserve"><flowRegion
|
||||
id="flowRegion4600"><rect
|
||||
y="415.4129"
|
||||
x="-38.183765"
|
||||
height="48.08326"
|
||||
width="257.38687"
|
||||
id="rect4602" /></flowRegion><flowPara
|
||||
id="flowPara4604"></flowPara></flowRoot> <text
|
||||
transform="scale(0.86288797,1.158899)"
|
||||
id="text4777"
|
||||
y="110.93711"
|
||||
x="0.93061"
|
||||
style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:28.14887619px;line-height:4.25;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Bold';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#003dff;fill-opacity:1;stroke:none;stroke-width:7.51955223;stroke-miterlimit:4;stroke-dasharray:none"
|
||||
xml:space="preserve"><tspan
|
||||
style="stroke-width:7.51955223"
|
||||
id="tspan4775"
|
||||
y="110.93711"
|
||||
x="0.93061"><tspan
|
||||
id="tspan4773"
|
||||
style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:28.14887619px;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Bold';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:3.56786728px;writing-mode:lr-tb;text-anchor:start;fill:#003dff;fill-opacity:1;stroke-width:7.51955223;stroke-miterlimit:4;stroke-dasharray:none"
|
||||
y="110.93711"
|
||||
x="0.93061">waybackpy</tspan></tspan></text>
|
||||
<rect
|
||||
y="98.311821"
|
||||
x="1.4967092"
|
||||
height="4.8643045"
|
||||
width="153.78688"
|
||||
id="rect4644"
|
||||
style="opacity:1;fill:#000080;fill-opacity:1;stroke:#00ff00;stroke-width:0;stroke-miterlimit:4;stroke-dasharray:none" />
|
||||
<rect
|
||||
style="opacity:1;fill:#000080;fill-opacity:1;stroke:#00ff00;stroke-width:0;stroke-miterlimit:4;stroke-dasharray:none"
|
||||
id="rect4648"
|
||||
width="153.78688"
|
||||
height="4.490128"
|
||||
x="23.573174"
|
||||
y="135.72957" />
|
||||
<rect
|
||||
y="135.72957"
|
||||
x="0.74835336"
|
||||
height="4.4901319"
|
||||
width="22.82482"
|
||||
id="rect4650"
|
||||
style="opacity:1;fill:#ff00ff;fill-opacity:1;stroke:#00ff00;stroke-width:0;stroke-miterlimit:4;stroke-dasharray:none" />
|
||||
<rect
|
||||
style="opacity:1;fill:#ff00ff;fill-opacity:1;stroke:#00ff00;stroke-width:0;stroke-miterlimit:4;stroke-dasharray:none"
|
||||
id="rect4652"
|
||||
width="21.702286"
|
||||
height="4.8643003"
|
||||
x="155.2836"
|
||||
y="98.311821" />
|
||||
</g>
|
||||
</svg>
|
After Width: | Height: | Size: 3.6 KiB |
525
index.rst
525
index.rst
@ -1,525 +0,0 @@
|
||||
waybackpy
|
||||
=========
|
||||
|
||||
|contributions welcome| |Build Status| |codecov| |Downloads| |Release|
|
||||
|Codacy Badge| |Maintainability| |CodeFactor| |made-with-python| |pypi|
|
||||
|PyPI - Python Version| |Maintenance| |Repo size| |License: MIT|
|
||||
|
||||
.. figure:: https://raw.githubusercontent.com/akamhy/waybackpy/master/assets/waybackpy-colored%20284.png
|
||||
:alt: Wayback Machine
|
||||
|
||||
Wayback Machine
|
||||
Waybackpy is a Python package that interfaces with `Internet
|
||||
Archive <https://en.wikipedia.org/wiki/Internet_Archive>`__'s `Wayback
|
||||
Machine <https://en.wikipedia.org/wiki/Wayback_Machine>`__ API. Archive
|
||||
webpages and retrieve archived webpages easily.
|
||||
|
||||
Table of contents
|
||||
=================
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<!--ts-->
|
||||
|
||||
- `Installation <#installation>`__
|
||||
|
||||
- `Usage <#usage>`__
|
||||
- `As a Python package <#as-a-python-package>`__
|
||||
|
||||
- `Saving a webpage <#capturing-aka-saving-an-url-using-save>`__
|
||||
- `Retrieving
|
||||
archive <#retrieving-the-archive-for-an-url-using-archive_url>`__
|
||||
- `Retrieving the oldest
|
||||
archive <#retrieving-the-oldest-archive-for-an-url-using-oldest>`__
|
||||
- `Retrieving the recent most/newest
|
||||
archive <#retrieving-the-newest-archive-for-an-url-using-newest>`__
|
||||
- `Retrieving the JSON response of availability
|
||||
API <#retrieving-the-json-reponse-for-the-avaliblity-api-request>`__
|
||||
- `Retrieving archive close to a specified year, month, day, hour,
|
||||
and
|
||||
minute <#retrieving-archive-close-to-a-specified-year-month-day-hour-and-minute-using-near>`__
|
||||
- `Get the content of
|
||||
webpage <#get-the-content-of-webpage-using-get>`__
|
||||
- `Count total archives for an
|
||||
URL <#count-total-archives-for-an-url-using-total_archives>`__
|
||||
- `List of URLs that Wayback Machine knows and has archived for a
|
||||
domain
|
||||
name <#list-of-urls-that-wayback-machine-knows-and-has-archived-for-a-domain-name>`__
|
||||
|
||||
- `With the Command-line
|
||||
interface <#with-the-command-line-interface>`__
|
||||
|
||||
- `Saving webpage <#save>`__
|
||||
- `Archive URL <#get-archive-url>`__
|
||||
- `Oldest archive URL <#oldest-archive>`__
|
||||
- `Newest archive URL <#newest-archive>`__
|
||||
- `JSON response of API <#get-json-data-of-avaialblity-api>`__
|
||||
- `Total archives <#total-number-of-archives>`__
|
||||
- `Archive near specified time <#archive-near-time>`__
|
||||
- `Get the source code <#get-the-source-code>`__
|
||||
- `Fetch all the URLs that the Wayback Machine knows for a
|
||||
domain <#fetch-all-the-urls-that-the-wayback-machine-knows-for-a-domain>`__
|
||||
|
||||
- `Tests <#tests>`__
|
||||
|
||||
- `Dependency <#dependency>`__
|
||||
|
||||
- `Packaging <#packaging>`__
|
||||
|
||||
- `License <#license>`__
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<!--te-->
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
Using `pip <https://en.wikipedia.org/wiki/Pip_(package_manager)>`__:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
pip install waybackpy
|
||||
|
||||
or direct from this repository using git.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
pip install git+https://github.com/akamhy/waybackpy.git
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
As a Python package
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Capturing aka Saving an url using save()
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: python
|
||||
|
||||
import waybackpy
|
||||
|
||||
url = "https://en.wikipedia.org/wiki/Multivariable_calculus"
|
||||
user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"
|
||||
|
||||
waybackpy_url_obj = waybackpy.Url(url, user_agent)
|
||||
archive = waybackpy_url_obj.save()
|
||||
print(archive)
|
||||
|
||||
.. code:: bash
|
||||
|
||||
https://web.archive.org/web/20201016171808/https://en.wikipedia.org/wiki/Multivariable_calculus
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPySaveExample\
|
||||
|
||||
Retrieving the archive for an URL using archive\_url
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: python
|
||||
|
||||
import waybackpy
|
||||
|
||||
url = "https://www.google.com/"
|
||||
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:40.0) Gecko/20100101 Firefox/40.0"
|
||||
|
||||
waybackpy_url_obj = waybackpy.Url(url, user_agent)
|
||||
archive_url = waybackpy_url_obj.archive_url
|
||||
print(archive_url)
|
||||
|
||||
.. code:: bash
|
||||
|
||||
https://web.archive.org/web/20201016153320/https://www.google.com/
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyArchiveUrl\
|
||||
|
||||
Retrieving the oldest archive for an URL using oldest()
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: python
|
||||
|
||||
import waybackpy
|
||||
|
||||
url = "https://www.google.com/"
|
||||
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:40.0) Gecko/20100101 Firefox/40.0"
|
||||
|
||||
waybackpy_url_obj = waybackpy.Url(url, user_agent)
|
||||
oldest_archive_url = waybackpy_url_obj.oldest()
|
||||
print(oldest_archive_url)
|
||||
|
||||
.. code:: bash
|
||||
|
||||
http://web.archive.org/web/19981111184551/http://google.com:80/
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyOldestExample\
|
||||
|
||||
Retrieving the newest archive for an URL using newest()
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: python
|
||||
|
||||
import waybackpy
|
||||
|
||||
url = "https://www.facebook.com/"
|
||||
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
|
||||
|
||||
waybackpy_url_obj = waybackpy.Url(url, user_agent)
|
||||
newest_archive_url = waybackpy_url_obj.newest()
|
||||
print(newest_archive_url)
|
||||
|
||||
.. code:: bash
|
||||
|
||||
https://web.archive.org/web/20201016150543/https://www.facebook.com/
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyNewestExample\
|
||||
|
||||
Retrieving the JSON reponse for the avaliblity API request
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: python
|
||||
|
||||
import waybackpy
|
||||
|
||||
url = "https://www.facebook.com/"
|
||||
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
|
||||
|
||||
waybackpy_url_obj = waybackpy.Url(url, user_agent)
|
||||
json_dict = waybackpy_url_obj.JSON
|
||||
print(json_dict)
|
||||
|
||||
.. code:: javascript
|
||||
|
||||
{'url': 'https://www.facebook.com/', 'archived_snapshots': {'closest': {'available': True, 'url': 'http://web.archive.org/web/20201016150543/https://www.facebook.com/', 'timestamp': '20201016150543', 'status': '200'}}}
|
||||
|
||||
Try this out in your browser @ https://repl.it/@akamhy/WaybackPyJSON\
|
||||
|
||||
Retrieving archive close to a specified year, month, day, hour, and minute using near()
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: python
|
||||
|
||||
from waybackpy import Url
|
||||
|
||||
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Firefox/38.0"
|
||||
url = "https://github.com/"
|
||||
|
||||
waybackpy_url_obj = Url(url, user_agent)
|
||||
|
||||
# Do not pad (don't use zeros in the month, year, day, minute, and hour arguments). e.g. For January, set month = 1 and not month = 01.
|
||||
|
||||
.. code:: python
|
||||
|
||||
github_archive_near_2010 = waybackpy_url_obj.near(year=2010)
|
||||
print(github_archive_near_2010)
|
||||
|
||||
.. code:: bash
|
||||
|
||||
https://web.archive.org/web/20101018053604/http://github.com:80/
|
||||
|
||||
.. code:: python
|
||||
|
||||
github_archive_near_2011_may = waybackpy_url_obj.near(year=2011, month=5)
|
||||
print(github_archive_near_2011_may)
|
||||
|
||||
.. code:: bash
|
||||
|
||||
https://web.archive.org/web/20110518233639/https://github.com/
|
||||
|
||||
.. code:: python
|
||||
|
||||
github_archive_near_2015_january_26 = waybackpy_url_obj.near(year=2015, month=1, day=26)
|
||||
print(github_archive_near_2015_january_26)
|
||||
|
||||
.. code:: bash
|
||||
|
||||
https://web.archive.org/web/20150125102636/https://github.com/
|
||||
|
||||
.. code:: python
|
||||
|
||||
github_archive_near_2018_4_july_9_2_am = waybackpy_url_obj.near(year=2018, month=7, day=4, hour=9, minute=2)
|
||||
print(github_archive_near_2018_4_july_9_2_am)
|
||||
|
||||
.. code:: bash
|
||||
|
||||
https://web.archive.org/web/20180704090245/https://github.com/
|
||||
|
||||
The package doesn't support second argument yet. You are encourged to
|
||||
create a PR ;)
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyNearExample\
|
||||
|
||||
Get the content of webpage using get()
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: python
|
||||
|
||||
import waybackpy
|
||||
|
||||
google_url = "https://www.google.com/"
|
||||
|
||||
User_Agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36"
|
||||
|
||||
waybackpy_url_object = waybackpy.Url(google_url, User_Agent)
|
||||
|
||||
|
||||
# If no argument is passed in get(), it gets the source of the Url used to create the object.
|
||||
current_google_url_source = waybackpy_url_object.get()
|
||||
print(current_google_url_source)
|
||||
|
||||
|
||||
# The following chunk of code will force a new archive of google.com and get the source of the archived page.
|
||||
# waybackpy_url_object.save() type is string.
|
||||
google_newest_archive_source = waybackpy_url_object.get(waybackpy_url_object.save())
|
||||
print(google_newest_archive_source)
|
||||
|
||||
|
||||
# waybackpy_url_object.oldest() type is str, it's oldest archive of google.com
|
||||
google_oldest_archive_source = waybackpy_url_object.get(waybackpy_url_object.oldest())
|
||||
print(google_oldest_archive_source)
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyGetExample#main.py\
|
||||
|
||||
Count total archives for an URL using total\_archives()
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: python
|
||||
|
||||
import waybackpy
|
||||
|
||||
URL = "https://en.wikipedia.org/wiki/Python (programming language)"
|
||||
UA = "Mozilla/5.0 (iPad; CPU OS 8_1_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B435 Safari/600.1.4"
|
||||
|
||||
waybackpy_url_object = waybackpy.Url(url=URL, user_agent=UA)
|
||||
|
||||
archive_count = waybackpy_url_object.total_archives()
|
||||
|
||||
print(archive_count) # total_archives() returns an int
|
||||
|
||||
.. code:: bash
|
||||
|
||||
2516
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyTotalArchivesExample\
|
||||
|
||||
List of URLs that Wayback Machine knows and has archived for a domain name
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
1) If alive=True is set, waybackpy will check all URLs to identify the
|
||||
alive URLs. Don't use with popular websites like google or it would
|
||||
take too long.
|
||||
2) To include URLs from subdomain set sundomain=True
|
||||
|
||||
.. code:: python
|
||||
|
||||
import waybackpy
|
||||
|
||||
URL = "akamhy.github.io"
|
||||
UA = "Mozilla/5.0 (iPad; CPU OS 8_1_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B435 Safari/600.1.4"
|
||||
|
||||
waybackpy_url_object = waybackpy.Url(url=URL, user_agent=UA)
|
||||
known_urls = waybackpy_url_object.known_urls(alive=True, subdomain=False) # alive and subdomain are optional.
|
||||
print(known_urls) # known_urls() returns list of URLs
|
||||
|
||||
.. code:: bash
|
||||
|
||||
['http://akamhy.github.io',
|
||||
'https://akamhy.github.io/waybackpy/',
|
||||
'https://akamhy.github.io/waybackpy/assets/css/style.css?v=a418a4e4641a1dbaad8f3bfbf293fad21a75ff11',
|
||||
'https://akamhy.github.io/waybackpy/assets/css/style.css?v=f881705d00bf47b5bf0c58808efe29eecba2226c']
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyKnownURLsToWayBackMachineExample#main.py\
|
||||
|
||||
With the Command-line interface
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Save
|
||||
^^^^
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ waybackpy --url "https://en.wikipedia.org/wiki/Social_media" --user_agent "my-unique-user-agent" --save
|
||||
https://web.archive.org/web/20200719062108/https://en.wikipedia.org/wiki/Social_media
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyBashSave\
|
||||
|
||||
Get archive URL
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ waybackpy --url "https://en.wikipedia.org/wiki/SpaceX" --user_agent "my-unique-user-agent" --archive_url
|
||||
https://web.archive.org/web/20201007132458/https://en.wikipedia.org/wiki/SpaceX
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyBashArchiveUrl\
|
||||
|
||||
Oldest archive
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ waybackpy --url "https://en.wikipedia.org/wiki/SpaceX" --user_agent "my-unique-user-agent" --oldest
|
||||
https://web.archive.org/web/20040803000845/http://en.wikipedia.org:80/wiki/SpaceX
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyBashOldest\
|
||||
|
||||
Newest archive
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ waybackpy --url "https://en.wikipedia.org/wiki/YouTube" --user_agent "my-unique-user-agent" --newest
|
||||
https://web.archive.org/web/20200606044708/https://en.wikipedia.org/wiki/YouTube
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyBashNewest\
|
||||
|
||||
Get JSON data of avaialblity API
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: bash
|
||||
|
||||
waybackpy --url "https://en.wikipedia.org/wiki/SpaceX" --user_agent "my-unique-user-agent" --json
|
||||
|
||||
.. code:: javascript
|
||||
|
||||
{'archived_snapshots': {'closest': {'timestamp': '20201007132458', 'status': '200', 'available': True, 'url': 'http://web.archive.org/web/20201007132458/https://en.wikipedia.org/wiki/SpaceX'}}, 'url': 'https://en.wikipedia.org/wiki/SpaceX'}
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyBashJSON\
|
||||
|
||||
Total number of archives
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ waybackpy --url "https://en.wikipedia.org/wiki/Linux_kernel" --user_agent "my-unique-user-agent" --total
|
||||
853
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyBashTotal\
|
||||
|
||||
Archive near time
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ waybackpy --url facebook.com --user_agent "my-unique-user-agent" --near --year 2012 --month 5 --day 12
|
||||
https://web.archive.org/web/20120512142515/https://www.facebook.com/
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyBashNear\
|
||||
|
||||
Get the source code
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: bash
|
||||
|
||||
waybackpy --url google.com --user_agent "my-unique-user-agent" --get url # Prints the source code of the url
|
||||
waybackpy --url google.com --user_agent "my-unique-user-agent" --get oldest # Prints the source code of the oldest archive
|
||||
waybackpy --url google.com --user_agent "my-unique-user-agent" --get newest # Prints the source code of the newest archive
|
||||
waybackpy --url google.com --user_agent "my-unique-user-agent" --get save # Save a new archive on wayback machine then print the source code of this archive.
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackPyBashGet\
|
||||
|
||||
Fetch all the URLs that the Wayback Machine knows for a domain
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
1) You can add the '--alive' flag to only fetch alive links.
|
||||
2) You can add the '--subdomain' flag to add subdomains.
|
||||
3) '--alive' and '--subdomain' flags can be used simultaneously.
|
||||
4) All links will be saved in a file, and the file will be created in
|
||||
the current working directory.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
pip install waybackpy
|
||||
|
||||
# Ignore the above installation line.
|
||||
|
||||
waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls
|
||||
# Prints all known URLs under akamhy.github.io
|
||||
|
||||
|
||||
waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --alive
|
||||
# Prints all known URLs under akamhy.github.io which are still working and not dead links.
|
||||
|
||||
|
||||
waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain
|
||||
# Prints all known URLs under akamhy.github.io inclusing subdomain
|
||||
|
||||
|
||||
waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain --alive
|
||||
# Prints all known URLs under akamhy.github.io including subdomain which are not dead links and still alive.
|
||||
|
||||
Try this out in your browser @
|
||||
https://repl.it/@akamhy/WaybackpyKnownUrlsFromWaybackMachine#main.sh\
|
||||
|
||||
Tests
|
||||
-----
|
||||
|
||||
`Here <https://github.com/akamhy/waybackpy/tree/master/tests>`__
|
||||
|
||||
To run tests locally:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
pip install -U pytest
|
||||
pip install codecov
|
||||
pip install pytest pytest-cov
|
||||
cd tests
|
||||
pytest --cov=../waybackpy
|
||||
python -m codecov #For reporting coverage on Codecov
|
||||
|
||||
Packaging
|
||||
---------
|
||||
|
||||
1. Increment version.
|
||||
|
||||
2. Build package ``python setup.py sdist bdist_wheel``.
|
||||
|
||||
3. Sign & upload the package ``twine upload -s dist/*``.
|
||||
|
||||
License
|
||||
-------
|
||||
|
||||
Released under the MIT License. See
|
||||
`license <https://github.com/akamhy/waybackpy/blob/master/LICENSE>`__
|
||||
for details.
|
||||
|
||||
.. |contributions welcome| image:: https://img.shields.io/static/v1.svg?label=Contributions&message=Welcome&color=0059b3&style=flat-square
|
||||
.. |Build Status| image:: https://img.shields.io/travis/akamhy/waybackpy.svg?label=Travis%20CI&logo=travis&style=flat-square
|
||||
:target: https://travis-ci.org/akamhy/waybackpy
|
||||
.. |codecov| image:: https://codecov.io/gh/akamhy/waybackpy/branch/master/graph/badge.svg
|
||||
:target: https://codecov.io/gh/akamhy/waybackpy
|
||||
.. |Downloads| image:: https://pepy.tech/badge/waybackpy/month
|
||||
:target: https://pepy.tech/project/waybackpy/month
|
||||
.. |Release| image:: https://img.shields.io/github/v/release/akamhy/waybackpy.svg
|
||||
:target: https://github.com/akamhy/waybackpy/releases
|
||||
.. |Codacy Badge| image:: https://api.codacy.com/project/badge/Grade/255459cede9341e39436ec8866d3fb65
|
||||
:target: https://www.codacy.com/manual/akamhy/waybackpy?utm_source=github.com&utm_medium=referral&utm_content=akamhy/waybackpy&utm_campaign=Badge_Grade
|
||||
.. |Maintainability| image:: https://api.codeclimate.com/v1/badges/942f13d8177a56c1c906/maintainability
|
||||
:target: https://codeclimate.com/github/akamhy/waybackpy/maintainability
|
||||
.. |CodeFactor| image:: https://www.codefactor.io/repository/github/akamhy/waybackpy/badge
|
||||
:target: https://www.codefactor.io/repository/github/akamhy/waybackpy
|
||||
.. |made-with-python| image:: https://img.shields.io/badge/Made%20with-Python-1f425f.svg
|
||||
:target: https://www.python.org/
|
||||
.. |pypi| image:: https://img.shields.io/pypi/v/waybackpy.svg
|
||||
:target: https://pypi.org/project/waybackpy/
|
||||
.. |PyPI - Python Version| image:: https://img.shields.io/pypi/pyversions/waybackpy?style=flat-square
|
||||
.. |Maintenance| image:: https://img.shields.io/badge/Maintained%3F-yes-green.svg
|
||||
:target: https://github.com/akamhy/waybackpy/graphs/commit-activity
|
||||
.. |Repo size| image:: https://img.shields.io/github/repo-size/akamhy/waybackpy.svg?label=Repo%20size&style=flat-square
|
||||
.. |License: MIT| image:: https://img.shields.io/badge/License-MIT-yellow.svg
|
||||
:target: https://github.com/akamhy/waybackpy/blob/master/LICENSE
|
Loading…
Reference in New Issue
Block a user