Update setup.py

* Waybackpy save example on replit

* Oldest example

* Newest method replit link

* Near method example

* Get example

* Total archive method example

.travis.yml

@@ -0,0 +1,14 @@
+language: python
+python:
+  - "2.7"
+  - "3.6"
+  - "3.8"
+os: linux
+dist: xenial
+cache: pip
+install:
+  - pip install pytest
+before_script:
+    cd tests
+script:
+  - pytest test_1.py

.whitesource

@@ -0,0 +1,8 @@
+{
+  "checkRunSettings": {
+    "vulnerableCheckRunConclusionLevel": "failure"
+  },
+  "issueSettings": {
+    "minSeverityLevel": "LOW"
+  }
+}

README.md

@@ -1,2 +1,230 @@
-# pywayback
-A python wrapper for Internet Archive's Wayback Machine
+# waybackpy
+
+[![Build Status](https://img.shields.io/travis/akamhy/waybackpy.svg?label=Travis%20CI&logo=travis&style=flat-square)](https://travis-ci.org/akamhy/waybackpy)
+[![Downloads](https://img.shields.io/pypi/dm/waybackpy.svg)](https://pypistats.org/packages/waybackpy)
+[![Release](https://img.shields.io/github/v/release/akamhy/waybackpy.svg)](https://github.com/akamhy/waybackpy/releases)
+[![Codacy Badge](https://api.codacy.com/project/badge/Grade/255459cede9341e39436ec8866d3fb65)](https://www.codacy.com/manual/akamhy/waybackpy?utm_source=github.com&utm_medium=referral&utm_content=akamhy/waybackpy&utm_campaign=Badge_Grade)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/akamhy/waybackpy/blob/master/LICENSE)
+[![Maintainability](https://api.codeclimate.com/v1/badges/942f13d8177a56c1c906/maintainability)](https://codeclimate.com/github/akamhy/waybackpy/maintainability)
+[![CodeFactor](https://www.codefactor.io/repository/github/akamhy/waybackpy/badge)](https://www.codefactor.io/repository/github/akamhy/waybackpy)
+[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)
+![pypi](https://img.shields.io/pypi/v/waybackpy.svg)
+![PyPI - Python Version](https://img.shields.io/pypi/pyversions/waybackpy?style=flat-square)
+[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/akamhy/waybackpy/graphs/commit-activity)
+[![codecov](https://codecov.io/gh/akamhy/waybackpy/branch/master/graph/badge.svg)](https://codecov.io/gh/akamhy/waybackpy)
+![](https://img.shields.io/github/repo-size/akamhy/waybackpy.svg?label=Repo%20size&style=flat-square)
+![contributions welcome](https://img.shields.io/static/v1.svg?label=Contributions&message=Welcome&color=0059b3&style=flat-square)
+
+
+![Internet Archive](https://upload.wikimedia.org/wikipedia/commons/thumb/8/84/Internet_Archive_logo_and_wordmark.svg/84px-Internet_Archive_logo_and_wordmark.svg.png)
+![Wayback Machine](https://upload.wikimedia.org/wikipedia/commons/thumb/0/01/Wayback_Machine_logo_2010.svg/284px-Wayback_Machine_logo_2010.svg.png)
+
+Waybackpy is a Python library that interfaces with the [Internet Archive](https://en.wikipedia.org/wiki/Internet_Archive)'s [Wayback Machine](https://en.wikipedia.org/wiki/Wayback_Machine) API. Archive pages and retrieve archived pages easily.
+
+Table of contents
+=================
+<!--ts-->
+
+* [Installation](#installation)
+
+* [Usage](#usage)
+  * [Saving an url using save()](#capturing-aka-saving-an-url-using-save)
+  * [Receiving the oldest archive for an URL Using oldest()](#receiving-the-oldest-archive-for-an-url-using-oldest)
+  * [Receiving the recent most/newest archive for an URL using newest()](#receiving-the-newest-archive-for-an-url-using-newest)
+  * [Receiving archive close to a specified year, month, day, hour, and minute using near()](#receiving-archive-close-to-a-specified-year-month-day-hour-and-minute-using-near)
+  * [Get the content of webpage using get()](#get-the-content-of-webpage-using-get)
+  * [Count total archives for an URL using total_archives()](#count-total-archives-for-an-url-using-total_archives)
+
+
+* [Tests](#tests)
+
+* [Dependency](#dependency)
+
+* [License](#license)
+
+<!--te-->
+
+## Installation
+Using [pip](https://en.wikipedia.org/wiki/Pip_(package_manager)):
+```bash
+pip install waybackpy
+```
+
+
+## Usage
+
+#### Capturing aka Saving an url using save()
+```python
+import waybackpy
+
+new_archive_url = waybackpy.Url(
+
+    url = "https://en.wikipedia.org/wiki/Multivariable_calculus",
+    user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"
+    
+).save()
+
+print(new_archive_url)
+```
+```bash
+https://web.archive.org/web/20200504141153/https://github.com/akamhy/waybackpy
+```
+<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPySaveExample></sub>
+
+
+
+#### Receiving the oldest archive for an URL using oldest()
+```python
+import waybackpy
+
+oldest_archive_url = waybackpy.Url(
+
+    "https://www.google.com/",
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:40.0) Gecko/20100101 Firefox/40.0"
+    
+).oldest()
+
+print(oldest_archive_url)
+```
+```bash
+http://web.archive.org/web/19981111184551/http://google.com:80/
+```
+<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyOldestExample></sub>
+
+
+
+#### Receiving the newest archive for an URL using newest()
+```python
+import waybackpy
+
+newest_archive_url = waybackpy.Url(
+
+    "https://www.facebook.com/",
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
+    
+).newest()
+
+print(newest_archive_url)
+```
+```bash
+https://web.archive.org/web/20200714013225/https://www.facebook.com/
+```
+<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyNewestExample></sub>
+
+
+
+#### Receiving archive close to a specified year, month, day, hour, and minute using near()
+```python
+from waybackpy import Url
+
+user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Firefox/38.0"
+github_url = "https://github.com/"
+
+
+github_wayback_obj = Url(github_url, user_agent)
+
+# Do not pad (don't use zeros in the month, year, day, minute, and hour arguments). e.g. For January, set month = 1 and not month = 01.
+```
+```python
+github_archive_near_2010 = github_wayback_obj.near(year=2010)
+print(github_archive_near_2010)
+```
+```bash
+https://web.archive.org/web/20100719134402/http://github.com/
+```
+```python
+github_archive_near_2011_may = github_wayback_obj.near(year=2011, month=5)
+print(github_archive_near_2011_may)
+```
+```bash
+https://web.archive.org/web/20110519185447/https://github.com/
+```
+```python
+github_archive_near_2015_january_26 = github_wayback_obj.near(
+    year=2015, month=1, day=26
+)
+print(github_archive_near_2015_january_26)
+```
+```bash
+https://web.archive.org/web/20150127031159/https://github.com
+```
+```python
+github_archive_near_2018_4_july_9_2_am = github_wayback_obj.near(
+    year=2018, month=7, day=4, hour = 9, minute = 2
+)
+print(github_archive_near_2018_4_july_9_2_am)
+```
+```bash
+https://web.archive.org/web/20180704090245/https://github.com/
+
+```
+
+<sub>The library doesn't supports seconds yet. You are encourged to create a PR ;)</sub>
+
+<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyNearExample></sub>
+
+
+
+#### Get the content of webpage using get()
+```python
+import waybackpy
+
+google_url = "https://www.google.com/"
+
+User_Agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36"
+
+waybackpy_url_object = waybackpy.Url(google_url, User_Agent)
+
+
+# If no argument is passed in get(), it gets the source of the Url used to create the object.
+current_google_url_source = waybackpy_url_object.get()
+print(current_google_url_source)
+
+
+# The following chunk of code will force a new archive of google.com and get the source of the archived page.
+# waybackpy_url_object.save() type is string.
+google_newest_archive_source = waybackpy_url_object.get(
+    waybackpy_url_object.save()
+)
+print(google_newest_archive_source)
+
+
+# waybackpy_url_object.oldest() type is str, it's oldest archive of google.com
+google_oldest_archive_source = waybackpy_url_object.get(
+    waybackpy_url_object.oldest()
+)
+print(google_oldest_archive_source)
+```
+<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyGetExample#main.py></sub>
+
+
+#### Count total archives for an URL using total_archives()
+```python
+import waybackpy
+
+URL = "https://en.wikipedia.org/wiki/Python (programming language)"
+
+UA = "Mozilla/5.0 (iPad; CPU OS 8_1_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B435 Safari/600.1.4"
+
+archive_count = waybackpy.Url(
+    url=URL,
+    user_agent=UA
+).total_archives()
+
+print(archive_count) # total_archives() returns an int
+```
+```bash
+2440
+```
+<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyTotalArchivesExample></sub>
+
+## Tests
+* [Here](https://github.com/akamhy/waybackpy/tree/master/tests)
+
+
+## Dependency
+* None, just python standard libraries (re, json, urllib and datetime). Both python 2 and 3 are supported :)
+
+
+## License
+[MIT License](https://github.com/akamhy/waybackpy/blob/master/LICENSE)

_config.yml

@@ -0,0 +1 @@
+theme: jekyll-theme-cayman

index.rst

@@ -0,0 +1,294 @@
+waybackpy
+=========
+
+|Build Status| |Downloads| |Release| |Codacy Badge| |License: MIT|
+|Maintainability| |CodeFactor| |made-with-python| |pypi| |PyPI - Python
+Version| |Maintenance| |codecov| |image12| |contributions welcome|
+
+|Internet Archive| |Wayback Machine|
+
+Waybackpy is a Python library that interfaces with the `Internet
+Archive <https://en.wikipedia.org/wiki/Internet_Archive>`__'s `Wayback
+Machine <https://en.wikipedia.org/wiki/Wayback_Machine>`__ API. Archive
+pages and retrieve archived pages easily.
+
+Table of contents
+=================
+
+.. raw:: html
+
+   <!--ts-->
+
+-  `Installation <#installation>`__
+
+-  `Usage <#usage>`__
+-  `Saving an url using
+   save() <#capturing-aka-saving-an-url-using-save>`__
+-  `Receiving the oldest archive for an URL Using
+   oldest() <#receiving-the-oldest-archive-for-an-url-using-oldest>`__
+-  `Receiving the recent most/newest archive for an URL using
+   newest() <#receiving-the-newest-archive-for-an-url-using-newest>`__
+-  `Receiving archive close to a specified year, month, day, hour, and
+   minute using
+   near() <#receiving-archive-close-to-a-specified-year-month-day-hour-and-minute-using-near>`__
+-  `Get the content of webpage using
+   get() <#get-the-content-of-webpage-using-get>`__
+-  `Count total archives for an URL using
+   total\_archives() <#count-total-archives-for-an-url-using-total_archives>`__
+
+-  `Tests <#tests>`__
+
+-  `Dependency <#dependency>`__
+
+-  `License <#license>`__
+
+.. raw:: html
+
+   <!--te-->
+
+Installation
+------------
+
+Using `pip <https://en.wikipedia.org/wiki/Pip_(package_manager)>`__:
+
+.. code:: bash
+
+    pip install waybackpy
+
+Usage
+-----
+
+Capturing aka Saving an url using save()
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: python
+
+    import waybackpy
+
+    new_archive_url = waybackpy.Url(
+
+        url = "https://en.wikipedia.org/wiki/Multivariable_calculus",
+        user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"
+        
+    ).save()
+
+    print(new_archive_url)
+
+.. code:: bash
+
+    https://web.archive.org/web/20200504141153/https://github.com/akamhy/waybackpy
+
+Try this out in your browser @
+https://repl.it/@akamhy/WaybackPySaveExample\ 
+
+Receiving the oldest archive for an URL using oldest()
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: python
+
+    import waybackpy
+
+    oldest_archive_url = waybackpy.Url(
+
+        "https://www.google.com/",
+        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:40.0) Gecko/20100101 Firefox/40.0"
+        
+    ).oldest()
+
+    print(oldest_archive_url)
+
+.. code:: bash
+
+    http://web.archive.org/web/19981111184551/http://google.com:80/
+
+Try this out in your browser @
+https://repl.it/@akamhy/WaybackPyOldestExample\ 
+
+Receiving the newest archive for an URL using newest()
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: python
+
+    import waybackpy
+
+    newest_archive_url = waybackpy.Url(
+
+        "https://www.facebook.com/",
+        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
+        
+    ).newest()
+
+    print(newest_archive_url)
+
+.. code:: bash
+
+    https://web.archive.org/web/20200714013225/https://www.facebook.com/
+
+Try this out in your browser @
+https://repl.it/@akamhy/WaybackPyNewestExample\ 
+
+Receiving archive close to a specified year, month, day, hour, and minute using near()
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: python
+
+    from waybackpy import Url
+
+    user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Firefox/38.0"
+    github_url = "https://github.com/"
+
+
+    github_wayback_obj = Url(github_url, user_agent)
+
+    # Do not pad (don't use zeros in the month, year, day, minute, and hour arguments). e.g. For January, set month = 1 and not month = 01.
+
+.. code:: python
+
+    github_archive_near_2010 = github_wayback_obj.near(year=2010)
+    print(github_archive_near_2010)
+
+.. code:: bash
+
+    https://web.archive.org/web/20100719134402/http://github.com/
+
+.. code:: python
+
+    github_archive_near_2011_may = github_wayback_obj.near(year=2011, month=5)
+    print(github_archive_near_2011_may)
+
+.. code:: bash
+
+    https://web.archive.org/web/20110519185447/https://github.com/
+
+.. code:: python
+
+    github_archive_near_2015_january_26 = github_wayback_obj.near(
+        year=2015, month=1, day=26
+    )
+    print(github_archive_near_2015_january_26)
+
+.. code:: bash
+
+    https://web.archive.org/web/20150127031159/https://github.com
+
+.. code:: python
+
+    github_archive_near_2018_4_july_9_2_am = github_wayback_obj.near(
+        year=2018, month=7, day=4, hour = 9, minute = 2
+    )
+    print(github_archive_near_2018_4_july_9_2_am)
+
+.. code:: bash
+
+    https://web.archive.org/web/20180704090245/https://github.com/
+
+The library doesn't supports seconds yet. You are encourged to create a
+PR ;)
+
+Try this out in your browser @
+https://repl.it/@akamhy/WaybackPyNearExample\ 
+
+Get the content of webpage using get()
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: python
+
+    import waybackpy
+
+    google_url = "https://www.google.com/"
+
+    User_Agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36"
+
+    waybackpy_url_object = waybackpy.Url(google_url, User_Agent)
+
+
+    # If no argument is passed in get(), it gets the source of the Url used to create the object.
+    current_google_url_source = waybackpy_url_object.get()
+    print(current_google_url_source)
+
+
+    # The following chunk of code will force a new archive of google.com and get the source of the archived page.
+    # waybackpy_url_object.save() type is string.
+    google_newest_archive_source = waybackpy_url_object.get(
+        waybackpy_url_object.save()
+    )
+    print(google_newest_archive_source)
+
+
+    # waybackpy_url_object.oldest() type is str, it's oldest archive of google.com
+    google_oldest_archive_source = waybackpy_url_object.get(
+        waybackpy_url_object.oldest()
+    )
+    print(google_oldest_archive_source)
+
+Try this out in your browser @
+https://repl.it/@akamhy/WaybackPyGetExample#main.py\ 
+
+Count total archives for an URL using total\_archives()
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: python
+
+    import waybackpy
+
+    URL = "https://en.wikipedia.org/wiki/Python (programming language)"
+
+    UA = "Mozilla/5.0 (iPad; CPU OS 8_1_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B435 Safari/600.1.4"
+
+    archive_count = waybackpy.Url(
+        url=URL,
+        user_agent=UA
+    ).total_archives()
+
+    print(archive_count) # total_archives() returns an int
+
+.. code:: bash
+
+    2440
+
+Try this out in your browser @
+https://repl.it/@akamhy/WaybackPyTotalArchivesExample\ 
+
+Tests
+-----
+
+-  `Here <https://github.com/akamhy/waybackpy/tree/master/tests>`__
+
+Dependency
+----------
+
+-  None, just python standard libraries (re, json, urllib and datetime).
+   Both python 2 and 3 are supported :)
+
+License
+-------
+
+`MIT
+License <https://github.com/akamhy/waybackpy/blob/master/LICENSE>`__
+
+.. |Build Status| image:: https://img.shields.io/travis/akamhy/waybackpy.svg?label=Travis%20CI&logo=travis&style=flat-square
+   :target: https://travis-ci.org/akamhy/waybackpy
+.. |Downloads| image:: https://img.shields.io/pypi/dm/waybackpy.svg
+   :target: https://pypistats.org/packages/waybackpy
+.. |Release| image:: https://img.shields.io/github/v/release/akamhy/waybackpy.svg
+   :target: https://github.com/akamhy/waybackpy/releases
+.. |Codacy Badge| image:: https://api.codacy.com/project/badge/Grade/255459cede9341e39436ec8866d3fb65
+   :target: https://www.codacy.com/manual/akamhy/waybackpy?utm_source=github.com&utm_medium=referral&utm_content=akamhy/waybackpy&utm_campaign=Badge_Grade
+.. |License: MIT| image:: https://img.shields.io/badge/License-MIT-yellow.svg
+   :target: https://github.com/akamhy/waybackpy/blob/master/LICENSE
+.. |Maintainability| image:: https://api.codeclimate.com/v1/badges/942f13d8177a56c1c906/maintainability
+   :target: https://codeclimate.com/github/akamhy/waybackpy/maintainability
+.. |CodeFactor| image:: https://www.codefactor.io/repository/github/akamhy/waybackpy/badge
+   :target: https://www.codefactor.io/repository/github/akamhy/waybackpy
+.. |made-with-python| image:: https://img.shields.io/badge/Made%20with-Python-1f425f.svg
+   :target: https://www.python.org/
+.. |pypi| image:: https://img.shields.io/pypi/v/waybackpy.svg
+.. |PyPI - Python Version| image:: https://img.shields.io/pypi/pyversions/waybackpy?style=flat-square
+.. |Maintenance| image:: https://img.shields.io/badge/Maintained%3F-yes-green.svg
+   :target: https://github.com/akamhy/waybackpy/graphs/commit-activity
+.. |codecov| image:: https://codecov.io/gh/akamhy/waybackpy/branch/master/graph/badge.svg
+   :target: https://codecov.io/gh/akamhy/waybackpy
+.. |image12| image:: https://img.shields.io/github/repo-size/akamhy/waybackpy.svg?label=Repo%20size&style=flat-square
+.. |contributions welcome| image:: https://img.shields.io/static/v1.svg?label=Contributions&message=Welcome&color=0059b3&style=flat-square
+.. |Internet Archive| image:: https://upload.wikimedia.org/wikipedia/commons/thumb/8/84/Internet_Archive_logo_and_wordmark.svg/84px-Internet_Archive_logo_and_wordmark.svg.png
+.. |Wayback Machine| image:: https://upload.wikimedia.org/wikipedia/commons/thumb/0/01/Wayback_Machine_logo_2010.svg/284px-Wayback_Machine_logo_2010.svg.png

setup.cfg

@@ -0,0 +1,3 @@
+[metadata]
+description-file = README.md
+license_file = LICENSE

setup.py

@@ -0,0 +1,49 @@
+import os.path
+from setuptools import setup
+
+with open(os.path.join(os.path.dirname(__file__), 'README.md')) as f:
+    long_description = f.read()
+
+about = {}
+with open(os.path.join(os.path.dirname(__file__), 'waybackpy', '__version__.py')) as f:
+    exec(f.read(), about)
+    
+setup(
+    name = about['__title__'],
+    packages = ['waybackpy'],
+    version = about['__version__'],
+    description = about['__description__'],
+    long_description=long_description,
+    long_description_content_type='text/markdown',
+    license= about['__license__'],
+    author = about['__author__'],
+    author_email = about['__author_email__'],
+    url = about['__url__'],
+    download_url = 'https://github.com/akamhy/waybackpy/archive/2.1.2.tar.gz',
+    keywords = ['wayback', 'archive', 'archive website', 'wayback machine', 'Internet Archive'],
+    install_requires=[],
+    python_requires= ">=2.7",
+    classifiers=[
+        'Development Status :: 5 - Production/Stable',
+        'Intended Audience :: Developers',
+        'Natural Language :: English',
+        'Topic :: Software Development :: Build Tools',
+        'License :: OSI Approved :: MIT License',
+        'Programming Language :: Python',
+        'Programming Language :: Python :: 2',
+        'Programming Language :: Python :: 2.7',
+        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.2',
+        'Programming Language :: Python :: 3.3',
+        'Programming Language :: Python :: 3.4',
+        'Programming Language :: Python :: 3.5',
+        'Programming Language :: Python :: 3.6',
+        'Programming Language :: Python :: 3.7',
+        'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: Implementation :: CPython',
+        ],
+    project_urls={
+        'Documentation': 'https://waybackpy.readthedocs.io',
+        'Source': 'https://github.com/akamhy/waybackpy',
+    },
+)

tests/test_1.py

@@ -0,0 +1,130 @@
+# -*- coding: utf-8 -*-
+import sys
+sys.path.append("..")
+import waybackpy
+import pytest
+import random
+import time
+if sys.version_info >= (3, 0):  # If the python ver >= 3
+    from urllib.request import Request, urlopen
+    from urllib.error import URLError
+else: # For python2.x
+    from urllib2 import Request, urlopen, URLError
+
+user_agent = "Mozilla/5.0 (Windows NT 6.2; rv:20.0) Gecko/20121202 Firefox/20.0"
+
+def test_clean_url():
+    time.sleep(10)
+    test_url = " https://en.wikipedia.org/wiki/Network security "
+    answer = "https://en.wikipedia.org/wiki/Network_security"
+    target = waybackpy.Url(test_url, user_agent)
+    test_result = target.clean_url()
+    assert answer == test_result
+
+def test_url_check():
+    time.sleep(10)
+    broken_url = "http://wwwgooglecom/"
+    with pytest.raises(Exception) as e_info:
+        waybackpy.Url(broken_url, user_agent)
+
+def test_save():
+    # Test for urls that exist and can be archived.
+    time.sleep(10)
+
+    url_list = [
+        "en.wikipedia.org",
+        "www.wikidata.org",
+        "commons.wikimedia.org",
+        "www.wiktionary.org",
+        "www.w3schools.com",
+        "www.youtube.com"
+    ]
+    x = random.randint(0, len(url_list)-1) 
+    url1 = url_list[x]
+    target = waybackpy.Url(url1, "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36")
+    archived_url1 = target.save()
+    assert url1 in archived_url1
+
+    if sys.version_info > (3, 6):
+
+        # Test for urls that are incorrect.
+        with pytest.raises(Exception) as e_info:
+            url2 = "ha ha ha ha"
+            waybackpy.Url(url2, user_agent)
+        time.sleep(5)
+        # Test for urls not allowed to archive by robot.txt.
+        with pytest.raises(Exception) as e_info:
+            url3 = "http://www.archive.is/faq.html"
+            target = waybackpy.Url(url3, "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:25.0) Gecko/20100101 Firefox/25.0")
+            target.save()
+
+        time.sleep(5)
+        # Non existent urls, test
+        with pytest.raises(Exception) as e_info:
+            url4 = "https://githfgdhshajagjstgeths537agajaajgsagudadhuss8762346887adsiugujsdgahub.us"
+            target = waybackpy.Url(url3, "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27")
+            target.save()
+
+    else:
+        pass
+
+def test_near():
+    time.sleep(10)
+    url = "google.com"
+    target = waybackpy.Url(url, "Mozilla/5.0 (Windows; U; Windows NT 6.0; de-DE) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4")
+    archive_near_year = target.near(year=2010)
+    assert "2010" in archive_near_year
+
+    if sys.version_info > (3, 6):
+        time.sleep(5)
+        archive_near_month_year = target.near( year=2015, month=2)
+        assert ("201502" in archive_near_month_year) or ("201501" in archive_near_month_year) or ("201503" in archive_near_month_year)
+    
+        target = waybackpy.Url("www.python.org", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246")
+        archive_near_hour_day_month_year = target.near(year=2008, month=5, day=9, hour=15)
+        assert ("2008050915" in archive_near_hour_day_month_year) or ("2008050914" in archive_near_hour_day_month_year) or ("2008050913" in archive_near_hour_day_month_year)
+    
+        with pytest.raises(Exception) as e_info:
+            NeverArchivedUrl = "https://ee_3n.wrihkeipef4edia.org/rwti5r_ki/Nertr6w_rork_rse7c_urity"
+            target = waybackpy.Url(NeverArchivedUrl, user_agent)
+            target.near(year=2010)
+    else:
+        pass
+
+def test_oldest():
+    time.sleep(10)
+    url = "github.com/akamhy/waybackpy"
+    target = waybackpy.Url(url, user_agent)
+    assert "20200504141153" in target.oldest()
+
+def test_newest():
+    time.sleep(10)
+    url = "github.com/akamhy/waybackpy"
+    target = waybackpy.Url(url, user_agent)
+    assert url in target.newest()
+
+def test_get():
+    time.sleep(10)
+    target = waybackpy.Url("google.com", user_agent)
+    assert "Welcome to Google" in target.get(target.oldest())
+
+def test_wayback_timestamp():
+    ts = waybackpy.Url("https://www.google.com","UA").wayback_timestamp(year=2020,month=1,day=2,hour=3,minute=4)
+    assert "202001020304" in str(ts)
+
+def test_get_response():
+    hdr = { 'User-Agent' : 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0'}
+    req = Request("https://www.google.com", headers=hdr) # nosec
+    response = waybackpy.Url("https://www.google.com","UA").get_response(req)
+    assert response.code == 200
+
+def test_total_archives():
+    time.sleep(10)
+    if sys.version_info > (3, 6):
+        target = waybackpy.Url(" https://google.com ", user_agent)
+        assert target.total_archives() > 500000
+    else:
+        pass
+    time.sleep(5)
+    target = waybackpy.Url(" https://gaha.e4i3n.m5iai3kip6ied.cima/gahh2718gs/ahkst63t7gad8 ", user_agent)
+    assert target.total_archives() == 0

waybackpy/__init__.py

@@ -1,6 +1,32 @@
 # -*- coding: utf-8 -*-
-from .wrapper import save, near, oldest, newest
 
-__version__ = "1.1"
+# ┏┓┏┓┏┓━━━━━━━━━━┏━━┓━━━━━━━━━━┏┓━━┏━━━┓━━━━━
+# ┃┃┃┃┃┃━━━━━━━━━━┃┏┓┃━━━━━━━━━━┃┃━━┃┏━┓┃━━━━━
+# ┃┃┃┃┃┃┏━━┓━┏┓━┏┓┃┗┛┗┓┏━━┓━┏━━┓┃┃┏┓┃┗━┛┃┏┓━┏┓
+# ┃┗┛┗┛┃┗━┓┃━┃┃━┃┃┃┏━┓┃┗━┓┃━┃┏━┛┃┗┛┛┃┏━━┛┃┃━┃┃
+# ┗┓┏┓┏┛┃┗┛┗┓┃┗━┛┃┃┗━┛┃┃┗┛┗┓┃┗━┓┃┏┓┓┃┃━━━┃┗━┛┃
+# ━┗┛┗┛━┗━━━┛┗━┓┏┛┗━━━┛┗━━━┛┗━━┛┗┛┗┛┗┛━━━┗━┓┏┛
+# ━━━━━━━━━━━┏━┛┃━━━━━━━━━━━━━━━━━━━━━━━━┏━┛┃━
+# ━━━━━━━━━━━┗━━┛━━━━━━━━━━━━━━━━━━━━━━━━┗━━┛━
 
-__all__ = ['wrapper', 'exceptions']
+"""
+Waybackpy is a Python library that interfaces with the Internet Archive's Wayback Machine API.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Archive pages and retrieve archived pages easily.
+
+Usage:
+   >>> import waybackpy
+   >>> target_url = waybackpy.Url('https://www.python.org', 'Your-apps-cool-user-agent')
+   >>> new_archive = target_url.save()
+   >>> print(new_archive)
+   https://web.archive.org/web/20200502170312/https://www.python.org/
+
+Full documentation @ <https://akamhy.github.io/waybackpy/>.
+:copyright: (c) 2020 by akamhy.
+:license: MIT
+"""
+
+from .wrapper import Url
+from .__version__ import __title__, __description__, __url__, __version__
+from .__version__ import __author__, __author_email__, __license__, __copyright__

waybackpy/__version__.py

@@ -0,0 +1,10 @@
+# -*- coding: utf-8 -*-
+
+__title__ = "waybackpy"
+__description__ = "A Python library that interfaces with the Internet Archive's Wayback Machine API. Archive pages and retrieve archived pages easily."
+__url__ = "https://akamhy.github.io/waybackpy/"
+__version__ = "2.1.2"
+__author__ = "akamhy"
+__author_email__ = "akash3pro@gmail.com"
+__license__ = "MIT"
+__copyright__ = "Copyright 2020 akamhy"

waybackpy/exceptions.py

@@ -1,38 +1,6 @@
 # -*- coding: utf-8 -*-
 
-class TooManyArchivingRequests(Exception):
+class WaybackError(Exception):
     """
-    Error when a single url reqeusted for archiving too many times in a short timespam.
-    Wayback machine doesn't supports archivng any url too many times in a short period of time.
-    """
-
-class ArchivingNotAllowed(Exception):
-    """
-    Files like robots.txt are set to deny robot archiving.
-    Wayback machine respects these file, will not archive.
-    """
-
-class PageNotSaved(Exception):
-    """
-    When unable to save a webpage.
-    """
-
-class ArchiveNotFound(Exception):
-    """
-    When a page was never archived but client asks for old archive.
-    """
-
-class UrlNotFound(Exception):
-    """
-    Raised when 404 UrlNotFound.
-    """
-
-class BadGateWay(Exception):
-    """
-    Raised when 502 bad gateway.
-    """
-
-class InvalidUrl(Exception):
-    """
-    Raised when url doesn't follow the standard url format.
+    Raised when API Service error.
     """

waybackpy/wrapper.py

@@ -1,68 +1,151 @@
 # -*- coding: utf-8 -*-
+
+import re
+import sys
+import json
 from datetime import datetime
-from waybackpy.exceptions import *
-try:
+from waybackpy.exceptions import WaybackError
+
+if sys.version_info >= (3, 0):  # If the python ver >= 3
     from urllib.request import Request, urlopen
-    from urllib.error import HTTPError
-except ImportError:
-    from urllib2 import Request, urlopen, HTTPError
+    from urllib.error import URLError
+else: # For python2.x
+    from urllib2 import Request, urlopen, URLError
+
+default_UA = "waybackpy python package - https://github.com/akamhy/waybackpy"
+
+class Url():
+    """waybackpy Url object"""
 
 
-default_UA = "waybackpy python package"
+    def __init__(self, url, user_agent=default_UA):
+        self.url = url
+        self.user_agent = user_agent
+        self.url_check() # checks url validity on init.
 
-def clean_url(url):
-    return str(url).strip().replace(" ","_")
+    def __repr__(self):
+        """Representation of the object."""
+        return "waybackpy.Url(url=%s, user_agent=%s)" % (self.url, self.user_agent)
 
-def save(url,UA=default_UA):
-    base_save_url = "https://web.archive.org/save/"
-    request_url = (base_save_url + clean_url(url))
-    hdr = { 'User-Agent' : '%s' % UA }
-    req = Request(request_url, headers=hdr)
-    if "." not in url:
-        raise InvalidUrl("'%s' is not a vaild url." % url)
-    try:
-        response = urlopen(req) #nosec
-    except HTTPError as e:
-        if e.code == 502:
-            raise BadGateWay(e)
-        elif e.code == 429:
-            raise TooManyArchivingRequests(e)
-        elif e.code == 404:
-            raise UrlNotFound(e)
-        else:
-          raise PageNotSaved(e)
+    def __str__(self):
+        """String representation of the object."""
+        return "%s" % self.clean_url()
 
-    header = response.headers
-    if "exclusion.robots.policy" in str(header):
-        raise ArchivingNotAllowed("Can not archive %s. Disabled by site owner." % (url))
-    archive_id = header['Content-Location']
-    archived_url = "https://web.archive.org" + archive_id
-    return archived_url
+    def __len__(self):
+        """Length of the URL."""
+        return len(self.clean_url())
 
-def near(
-    url,
-    year=datetime.utcnow().strftime('%Y'),
-    month=datetime.utcnow().strftime('%m'),
-    day=datetime.utcnow().strftime('%d'),
-    hour=datetime.utcnow().strftime('%H'),
-    minute=datetime.utcnow().strftime('%M'),
-    UA=default_UA,
-    ):
-    timestamp = str(year)+str(month)+str(day)+str(hour)+str(minute)
-    request_url = "https://archive.org/wayback/available?url=%s&timestamp=%s" % (clean_url(url), str(timestamp))
-    hdr = { 'User-Agent' : '%s' % UA }
-    req = Request(request_url, headers=hdr)
-    response = urlopen(req) #nosec
-    import json
-    data = json.loads(response.read().decode('utf8'))
-    if not data["archived_snapshots"]:
-        raise ArchiveNotFound("'%s' is not yet archived." % url)
-    
-    archive_url = (data["archived_snapshots"]["closest"]["url"])
-    return archive_url
+    def url_check(self):
+        """Check for common URL problems."""
+        if "." not in self.url:
+            raise URLError("'%s' is not a vaild url." % self.url)
+        return True
 
-def oldest(url,UA=default_UA,year=1994):
-    return near(url,year=year,UA=UA)
+    def clean_url(self):
+        """Fix the URL, if possible."""
+        return str(self.url).strip().replace(" ","_")
 
-def newest(url,UA=default_UA):
-    return near(url,UA=UA)
+    def wayback_timestamp(self, **kwargs):
+        """Return the formatted the timestamp."""
+        return (
+          str(kwargs["year"])
+          +
+          str(kwargs["month"]).zfill(2)
+          +
+          str(kwargs["day"]).zfill(2)
+          +
+          str(kwargs["hour"]).zfill(2)
+          +
+          str(kwargs["minute"]).zfill(2)
+          )
+
+    def save(self):
+        """Create a new archives for an URL on the Wayback Machine."""
+        request_url = ("https://web.archive.org/save/" + self.clean_url())
+        hdr = { 'User-Agent' : '%s' % self.user_agent } #nosec
+        req = Request(request_url, headers=hdr) #nosec
+        header = self.get_response(req).headers
+
+        def archive_url_parser(header):
+            """Parse out the archive from header."""
+            #Regex1
+            arch = re.search(r"rel=\"memento.*?(web\.archive\.org/web/[0-9]{14}/.*?)>", str(header))
+            if arch:
+                return arch.group(1)
+            #Regex2
+            arch = re.search(r"X-Cache-Key:\shttps(.*)[A-Z]{2}", str(header))
+            if arch:
+                return arch.group(1)
+            raise WaybackError(
+                "No archive url found in the API response. Visit https://github.com/akamhy/waybackpy for latest version of waybackpy.\nHeader:\n%s" % str(header)
+            )
+
+        return "https://" + archive_url_parser(header)
+
+    def get(self, url=None, user_agent=None, encoding=None):
+        """Returns the source code of the supplied URL. Auto detects the encoding if not supplied."""
+
+        if not url:
+            url = self.clean_url()
+        if not user_agent:
+            user_agent = self.user_agent
+
+        hdr = { 'User-Agent' : '%s' % user_agent }
+        req = Request(url, headers=hdr) #nosec
+        response = self.get_response(req)
+        if not encoding:
+            try:
+                encoding= response.headers['content-type'].split('charset=')[-1]
+            except AttributeError:
+                encoding = "UTF-8"
+        return response.read().decode(encoding.replace("text/html", "UTF-8", 1))
+
+    def get_response(self, req):
+        """Get response for the supplied request."""
+        try:
+            response = urlopen(req) #nosec
+        except Exception:
+            try:
+                 response = urlopen(req) #nosec
+            except Exception as e:
+                raise WaybackError(e)
+        return response
+
+    def near(self, **kwargs):
+        """ Returns the archived from Wayback Machine for an URL closest to the time supplied.
+            Supported params are year, month, day, hour and minute.
+            The non supplied parameters are default to the runtime time.
+        """
+        year=kwargs.get("year", datetime.utcnow().strftime('%Y'))
+        month=kwargs.get("month", datetime.utcnow().strftime('%m'))
+        day=kwargs.get("day", datetime.utcnow().strftime('%d'))
+        hour=kwargs.get("hour", datetime.utcnow().strftime('%H'))
+        minute=kwargs.get("minute", datetime.utcnow().strftime('%M'))
+        timestamp = self.wayback_timestamp(year=year,month=month,day=day,hour=hour,minute=minute)
+        request_url = "https://archive.org/wayback/available?url=%s&timestamp=%s" % (self.clean_url(), str(timestamp))
+        hdr = { 'User-Agent' : '%s' % self.user_agent }
+        req = Request(request_url, headers=hdr) # nosec
+        response = self.get_response(req)
+        data = json.loads(response.read().decode("UTF-8"))
+        if not data["archived_snapshots"]:
+            raise WaybackError("'%s' is not yet archived." % url)
+        archive_url = (data["archived_snapshots"]["closest"]["url"])
+        # wayback machine returns http sometimes, idk why? But they support https
+        archive_url = archive_url.replace("http://web.archive.org/web/","https://web.archive.org/web/",1)
+        return archive_url
+
+    def oldest(self, year=1994):
+        """Returns the oldest archive from Wayback Machine for an URL."""
+        return self.near(year=year)
+
+    def newest(self):
+        """Returns the newest archive on Wayback Machine for an URL, sometimes you may not get the newest archive because wayback machine DB lag."""
+        return self.near()
+
+    def total_archives(self):
+        """Returns the total number of archives on Wayback Machine for an URL."""
+        hdr = { 'User-Agent' : '%s' % self.user_agent }
+        request_url = "https://web.archive.org/cdx/search/cdx?url=%s&output=json&fl=statuscode" % self.clean_url()
+        req = Request(request_url, headers=hdr) # nosec
+        response = self.get_response(req)
+        return str(response.read()).count(",") # Most efficient method to count number of archives (yet)