No need to fetch archive_url and timestamp from availability API on init (#55 )

* No need to fetch archive_url and timestamp from availability API on init. Not useful if all I want is to archive a page * Update test_wrapper.py * Update wrapper.py * Update test_wrapper.py * Update wrapper.py * Update cli.py * Update wrapper.py * Update __version__.py * Update __version__.py * Update __version__.py * Update __version__.py * Update setup.py * Update README.md
v2.3.1 (#54 )
2021-01-02 11:10:23 +05:30 · 2021-01-01 19:15:23 +05:30 · 2021-01-01 16:38:57 +05:30 · 2020-12-15 10:24:13 +05:30 · 2020-12-14 01:18:04 +05:30 · 2020-12-14 00:16:29 +05:30
23 changed files with 1128 additions and 835 deletions
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@ -0,0 +1,42 @@
+# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: CI
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.8']
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install flake8 pytest codecov pytest-cov
+        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+    - name: Test with pytest
+      run: |
+        pytest --cov=waybackpy tests/
+    - name: Upload coverage to Codecov
+      run: |
+        bash <(curl -s https://codecov.io/bash) -t ${{ secrets.CODECOV_TOKEN }}
--- a/.gitignore
+++ b/.gitignore
@ -1,3 +1,6 @@
+# Files generated while testing
+*-urls-*.txt
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
--- a/.pep8speaks.yml
+++ b/.pep8speaks.yml
@ -0,0 +1,4 @@
+# File : .pep8speaks.yml
+
+scanner:
+    diff_only: True  # If True, errors caused by only the patch are shown
--- a/.pyup.yml
+++ b/.pyup.yml
@ -0,0 +1,5 @@
+# autogenerated pyup.io config file 
+# see https://pyup.io/docs/configuration/ for all available options
+
+schedule: ''
+update: false
--- a/.travis.yml
+++ b/.travis.yml
@ -1,19 +0,0 @@
-language: python
-os: linux
-dist: xenial
-cache: pip
-python:
-  - 2.7
-  - 3.6
-  - 3.8
-before_install:
-  - python --version
-  - pip install -U pip
-  - pip install -U pytest
-  - pip install codecov
-  - pip install pytest pytest-cov
-script:
-  - cd tests
-  - pytest --cov=../waybackpy
-after_success:
-  - if [[ $TRAVIS_PYTHON_VERSION == 3.8 ]]; then python -m codecov; fi
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@ -0,0 +1,58 @@
+# Contributing to waybackpy
+
+We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:
+
+- Reporting a bug
+- Discussing the current state of the code
+- Submitting a fix
+- Proposing new features
+- Becoming a maintainer
+
+## We Develop with Github
+
+We use github to host code, to track issues and feature requests, as well as accept pull requests.
+
+## We Use [Github Flow](https://guides.github.com/introduction/flow/index.html), So All Code Changes Happen Through Pull Requests
+
+Pull requests are the best way to propose changes to the codebase (we use [Github Flow](https://guides.github.com/introduction/flow/index.html)). We actively welcome your pull requests:
+
+1. Fork the repo and create your branch from `master`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. Issue that pull request!
+
+## Any contributions you make will be under the MIT Software License
+
+In short, when you submit code changes, your submissions are understood to be under the same [MIT License](https://github.com/akamhy/waybackpy/blob/master/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern.
+
+## Report bugs using Github's [issues](https://github.com/akamhy/waybackpy/issues)
+
+We use GitHub issues to track public bugs. Report a bug by [opening a new issue](https://github.com/akamhy/waybackpy/issues/new); it's that easy!
+
+## Write bug reports with detail, background, and sample code
+
+**Great Bug Reports** tend to have:
+
+- A quick summary and/or background
+- Steps to reproduce
+  - Be specific!
+  - Give sample code if you can.
+- What you expected would happen
+- What actually happens
+- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)
+
+People *love* thorough bug reports. I'm not even kidding.
+
+## Use a Consistent Coding Style
+
+* You can try running `flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics` for style unification.
+
+## License
+
+By contributing, you agree that your contributions will be licensed under its [MIT License](https://github.com/akamhy/waybackpy/blob/master/LICENSE).
+
+## References
+
+This document is forked from [this gist](https://gist.github.com/briandk/3d2e8b3ec8daf5a27a62) by [briandk](https://github.com/briandk) which was itself adapted from the open-source contribution guidelines for [Facebook's Draft](https://github.com/facebook/draft-js/blob/a9316a723f9e918afde44dea68b5f9f39b7d9b00/CONTRIBUTING.md)
--- a/2
+++ b/2
@ -1,6 +1,6 @@
 MIT License

-Copyright (c) 2020 akamhy
+Copyright (c) 2020 waybackpy contributors ( https://github.com/akamhy/waybackpy/graphs/contributors )

 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
--- a/README.md
+++ b/README.md
@ -1,23 +1,25 @@
-# waybackpy
+<div align="center">
+  <img src="https://raw.githubusercontent.com/akamhy/waybackpy/master/assets/waybackpy_logo.svg"><br>
+</div>

-![contributions welcome](https://img.shields.io/static/v1.svg?label=Contributions&message=Welcome&color=0059b3&style=flat-square)
-[![Build Status](https://img.shields.io/travis/akamhy/waybackpy.svg?label=Travis%20CI&logo=travis&style=flat-square)](https://travis-ci.org/akamhy/waybackpy)
-[![codecov](https://codecov.io/gh/akamhy/waybackpy/branch/master/graph/badge.svg)](https://codecov.io/gh/akamhy/waybackpy)
-[![Downloads](https://pepy.tech/badge/waybackpy/month)](https://pepy.tech/project/waybackpy/month)
-[![Release](https://img.shields.io/github/v/release/akamhy/waybackpy.svg)](https://github.com/akamhy/waybackpy/releases)
-[![Codacy Badge](https://api.codacy.com/project/badge/Grade/255459cede9341e39436ec8866d3fb65)](https://www.codacy.com/manual/akamhy/waybackpy?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=akamhy/waybackpy&amp;utm_campaign=Badge_Grade)
-[![Maintainability](https://api.codeclimate.com/v1/badges/942f13d8177a56c1c906/maintainability)](https://codeclimate.com/github/akamhy/waybackpy/maintainability)
-[![CodeFactor](https://www.codefactor.io/repository/github/akamhy/waybackpy/badge)](https://www.codefactor.io/repository/github/akamhy/waybackpy)
-[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)
+-----------------
+
+## Python package & CLI tool that interfaces with the Wayback Machine API. 
 [![pypi](https://img.shields.io/pypi/v/waybackpy.svg)](https://pypi.org/project/waybackpy/)
-![PyPI - Python Version](https://img.shields.io/pypi/pyversions/waybackpy?style=flat-square)
-[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/akamhy/waybackpy/graphs/commit-activity)
-![Repo size](https://img.shields.io/github/repo-size/akamhy/waybackpy.svg?label=Repo%20size&style=flat-square)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/akamhy/waybackpy/blob/master/LICENSE)
+[![Build Status](https://github.com/akamhy/waybackpy/workflows/CI/badge.svg)](https://github.com/akamhy/waybackpy/actions?query=workflow%3ACI)
+[![codecov](https://codecov.io/gh/akamhy/waybackpy/branch/master/graph/badge.svg)](https://codecov.io/gh/akamhy/waybackpy)
+[![contributions welcome](https://img.shields.io/static/v1.svg?label=Contributions&message=Welcome&color=0059b3&style=flat-square)](https://github.com/akamhy/waybackpy/blob/master/CONTRIBUTING.md)
+[![Codacy Badge](https://api.codacy.com/project/badge/Grade/255459cede9341e39436ec8866d3fb65)](https://www.codacy.com/manual/akamhy/waybackpy?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=akamhy/waybackpy&amp;utm_campaign=Badge_Grade)
+[![Downloads](https://pepy.tech/badge/waybackpy/month)](https://pepy.tech/project/waybackpy)
+[![Release](https://img.shields.io/github/v/release/akamhy/waybackpy.svg)](https://github.com/akamhy/waybackpy/releases)
+[![Maintainability](https://api.codeclimate.com/v1/badges/942f13d8177a56c1c906/maintainability)](https://codeclimate.com/github/akamhy/waybackpy/maintainability)
+[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)
+[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/akamhy/waybackpy/graphs/commit-activity)
+[![GitHub last commit](https://img.shields.io/github/last-commit/akamhy/waybackpy?color=blue&style=flat-square)](https://github.com/akamhy/waybackpy/commits/master)
+![PyPI - Python Version](https://img.shields.io/pypi/pyversions/waybackpy?style=flat-square)

-![Wayback Machine](https://raw.githubusercontent.com/akamhy/waybackpy/master/assets/waybackpy-colored%20284.png)

-Waybackpy is a Python library that interfaces with [Internet Archive](https://en.wikipedia.org/wiki/Internet_Archive)'s [Wayback Machine](https://en.wikipedia.org/wiki/Wayback_Machine) API. Archive webpages and retrieve archived webpages easily.

 Table of contents
 =================
@ -27,27 +29,29 @@ Table of contents

 * [Usage](#usage)
  * [As a Python package](#as-a-python-package)
-    * [Saving an url](#capturing-aka-saving-an-url-using-save)
+    * [Saving a webpage](#capturing-aka-saving-an-url-using-save)
+    * [Retrieving archive](#retrieving-the-archive-for-an-url-using-archive_url)
    * [Retrieving the oldest archive](#retrieving-the-oldest-archive-for-an-url-using-oldest)
-    * [Retrieving the recent most/newest archive](#retrieving-the-newest-archive-for-an-url-using-newest)
+    * [Retrieving the latest/newest archive](#retrieving-the-newest-archive-for-an-url-using-newest)
+    * [Retrieving the JSON response of availability API](#retrieving-the-json-response-for-the-availability-api-request)
    * [Retrieving archive close to a specified year, month, day, hour, and minute](#retrieving-archive-close-to-a-specified-year-month-day-hour-and-minute-using-near)
    * [Get the content of webpage](#get-the-content-of-webpage-using-get)
    * [Count total archives for an URL](#count-total-archives-for-an-url-using-total_archives)
-    * [List of URLs that Wayback Machine knows and has archived for a domain name](#retrieving-archive-close-to-a-specified-year-month-day-hour-and-minute-using-near)
+    * [List of URLs that Wayback Machine knows and has archived for a domain name](#list-of-urls-that-wayback-machine-knows-and-has-archived-for-a-domain-name)

-  * [As a Command-line tool](#with-the-command-line-interface)
-    * [Save](#save)
-    * [Oldest archive](#oldest-archive)
-    * [Newest archive](#newest-archive)
+  * [With the Command-line interface](#with-the-command-line-interface)
+    * [Saving webpage](#save)
+    * [Archive URL](#get-archive-url)
+    * [Oldest archive URL](#oldest-archive)
+    * [Newest archive URL](#newest-archive)
+    * [JSON response of API](#get-json-data-of-avaialblity-api)
    * [Total archives](#total-number-of-archives)
-    * [Archive near a time](#archive-near-time)
+    * [Archive near specified time](#archive-near-time)
    * [Get the source code](#get-the-source-code)
    * [Fetch all the URLs that the Wayback Machine knows for a domain](#fetch-all-the-urls-that-the-wayback-machine-knows-for-a-domain)

 * [Tests](#tests)

-* [Dependency](#dependency)
-
 * [Packaging](#packaging)

 * [License](#license)
@ -72,38 +76,54 @@ pip install git+https://github.com/akamhy/waybackpy.git

 ### As a Python package

-#### Capturing aka Saving an url using save()
+#### Capturing aka Saving an URL using save()

 ```python
 import waybackpy

-new_archive_url = waybackpy.Url(
+url = "https://en.wikipedia.org/wiki/Multivariable_calculus"
+user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"

-    url = "https://en.wikipedia.org/wiki/Multivariable_calculus",
-    user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"
-
-).save()
-
-print(new_archive_url)
+waybackpy_url_obj = waybackpy.Url(url, user_agent)
+archive = waybackpy_url_obj.save()
+print(archive)
 ```

 ```bash
-https://web.archive.org/web/20200504141153/https://github.com/akamhy/waybackpy
+https://web.archive.org/web/20201016171808/https://en.wikipedia.org/wiki/Multivariable_calculus
 ```

 <sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPySaveExample></sub>

+#### Retrieving the archive for an URL using archive_url
+
+```python
+import waybackpy
+
+url = "https://www.google.com/"
+user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:40.0) Gecko/20100101 Firefox/40.0"
+
+waybackpy_url_obj = waybackpy.Url(url, user_agent)
+archive_url = waybackpy_url_obj.archive_url
+print(archive_url)
+```
+
+```bash
+https://web.archive.org/web/20201016153320/https://www.google.com/
+```
+
+<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyArchiveUrl></sub>
+
 #### Retrieving the oldest archive for an URL using oldest()

 ```python
 import waybackpy

-oldest_archive_url = waybackpy.Url(
-
-    "https://www.google.com/",
-    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:40.0) Gecko/20100101 Firefox/40.0"
-).oldest()
+url = "https://www.google.com/"
+user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:40.0) Gecko/20100101 Firefox/40.0"

+waybackpy_url_obj = waybackpy.Url(url, user_agent)
+oldest_archive_url = waybackpy_url_obj.oldest()
 print(oldest_archive_url)
 ```

@ -118,78 +138,89 @@ http://web.archive.org/web/19981111184551/http://google.com:80/
 ```python
 import waybackpy

-newest_archive_url = waybackpy.Url(
-
-    "https://www.facebook.com/",
-    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
-
-).newest()
+url = "https://www.facebook.com/"
+user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"

+waybackpy_url_obj = waybackpy.Url(url, user_agent)
+newest_archive_url = waybackpy_url_obj.newest()
 print(newest_archive_url)
 ```

 ```bash
-https://web.archive.org/web/20200714013225/https://www.facebook.com/
+https://web.archive.org/web/20201016150543/https://www.facebook.com/
 ```

 <sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyNewestExample></sub>

+#### Retrieving the JSON response for the availability API request
+
+```python
+import waybackpy
+
+url = "https://www.facebook.com/"
+user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
+
+waybackpy_url_obj = waybackpy.Url(url, user_agent)
+json_dict = waybackpy_url_obj.JSON
+print(json_dict)
+```
+
+```javascript
+{'url': 'https://www.facebook.com/', 'archived_snapshots': {'closest': {'available': True, 'url': 'http://web.archive.org/web/20201016150543/https://www.facebook.com/', 'timestamp': '20201016150543', 'status': '200'}}}
+```
+
+<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyJSON></sub>
+
 #### Retrieving archive close to a specified year, month, day, hour, and minute using near()

 ```python
 from waybackpy import Url

 user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Firefox/38.0"
-github_url = "https://github.com/"
+url = "https://github.com/"

-
-github_wayback_obj = Url(github_url, user_agent)
+waybackpy_url_obj = Url(url, user_agent)

 # Do not pad (don't use zeros in the month, year, day, minute, and hour arguments). e.g. For January, set month = 1 and not month = 01.
 ```

 ```python
-github_archive_near_2010 = github_wayback_obj.near(year=2010)
+github_archive_near_2010 = waybackpy_url_obj.near(year=2010)
 print(github_archive_near_2010)
 ```

 ```bash
-https://web.archive.org/web/20100719134402/http://github.com/
+https://web.archive.org/web/20101018053604/http://github.com:80/
 ```

 ```python
-github_archive_near_2011_may = github_wayback_obj.near(year=2011, month=5)
+github_archive_near_2011_may = waybackpy_url_obj.near(year=2011, month=5)
 print(github_archive_near_2011_may)
 ```

 ```bash
-https://web.archive.org/web/20110519185447/https://github.com/
+https://web.archive.org/web/20110518233639/https://github.com/
 ```

 ```python
-github_archive_near_2015_january_26 = github_wayback_obj.near(
-    year=2015, month=1, day=26
-)
+github_archive_near_2015_january_26 = waybackpy_url_obj.near(year=2015, month=1, day=26)
 print(github_archive_near_2015_january_26)
 ```

 ```bash
-https://web.archive.org/web/20150127031159/https://github.com
+https://web.archive.org/web/20150125102636/https://github.com/
 ```

 ```python
-github_archive_near_2018_4_july_9_2_am = github_wayback_obj.near(
-    year=2018, month=7, day=4, hour = 9, minute = 2
-)
+github_archive_near_2018_4_july_9_2_am = waybackpy_url_obj.near(year=2018, month=7, day=4, hour=9, minute=2)
 print(github_archive_near_2018_4_july_9_2_am)
 ```

 ```bash
 https://web.archive.org/web/20180704090245/https://github.com/
-
 ```

-<sub>The library doesn't supports seconds yet. You are encourged to create a PR ;)</sub>
+<sub>The package doesn't support the seconds' argument yet. You are encouraged to create a PR ;)</sub>

 <sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyNearExample></sub>

@ -212,16 +243,12 @@ print(current_google_url_source)

 # The following chunk of code will force a new archive of google.com and get the source of the archived page.
 # waybackpy_url_object.save() type is string.
-google_newest_archive_source = waybackpy_url_object.get(
-    waybackpy_url_object.save()
-)
+google_newest_archive_source = waybackpy_url_object.get(waybackpy_url_object.save())
 print(google_newest_archive_source)


 # waybackpy_url_object.oldest() type is str, it's oldest archive of google.com
-google_oldest_archive_source = waybackpy_url_object.get(
-    waybackpy_url_object.oldest()
-)
+google_oldest_archive_source = waybackpy_url_object.get(waybackpy_url_object.oldest())
 print(google_oldest_archive_source)
 ```

@ -233,24 +260,22 @@ print(google_oldest_archive_source)
 import waybackpy

 URL = "https://en.wikipedia.org/wiki/Python (programming language)"
-
 UA = "Mozilla/5.0 (iPad; CPU OS 8_1_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B435 Safari/600.1.4"

-archive_count = waybackpy.Url(
-    url=URL,
-    user_agent=UA
-).total_archives()
+waybackpy_url_object = waybackpy.Url(url=URL, user_agent=UA)
+
+archive_count = waybackpy_url_object.total_archives()

 print(archive_count) # total_archives() returns an int
 ```

 ```bash
-2440
+2516
 ```

 <sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyTotalArchivesExample></sub>

-####  List of URLs that Wayback Machine knows and has archived for a domain name
+#### List of URLs that Wayback Machine knows and has archived for a domain name

 1) If alive=True is set, waybackpy will check all URLs to identify the alive URLs. Don't use with popular websites like google or it would take too long.
 2) To include URLs from subdomain set sundomain=True
@ -261,8 +286,8 @@ import waybackpy
 URL = "akamhy.github.io"
 UA = "Mozilla/5.0 (iPad; CPU OS 8_1_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B435 Safari/600.1.4"

-known_urls = waybackpy.Url(url=URL, user_agent=UA).known_urls(alive=True, subdomain=False) # alive and subdomain are optional.
-
+waybackpy_url_object = waybackpy.Url(url=URL, user_agent=UA)
+known_urls = waybackpy_url_object.known_urls(alive=True, subdomain=False) # alive and subdomain are optional.
 print(known_urls) # known_urls() returns list of URLs
 ```

@ -286,6 +311,15 @@ https://web.archive.org/web/20200719062108/https://en.wikipedia.org/wiki/Social_

 <sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyBashSave></sub>

+#### Get archive URL
+
+```bash
+$ waybackpy --url "https://en.wikipedia.org/wiki/SpaceX" --user_agent "my-unique-user-agent" --archive_url
+https://web.archive.org/web/20201007132458/https://en.wikipedia.org/wiki/SpaceX
+```
+
+<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyBashArchiveUrl></sub>
+
 #### Oldest archive

 ```bash
@ -304,6 +338,20 @@ https://web.archive.org/web/20200606044708/https://en.wikipedia.org/wiki/YouTube

 <sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyBashNewest></sub>

+#### Get JSON data of avaialblity API
+
+```bash
+waybackpy --url "https://en.wikipedia.org/wiki/SpaceX" --user_agent "my-unique-user-agent" --json
+
+```
+
+```javascript
+{'archived_snapshots': {'closest': {'timestamp': '20201007132458', 'status': '200', 'available': True, 'url': 'http://web.archive.org/web/20201007132458/https://en.wikipedia.org/wiki/SpaceX'}}, 'url': 'https://en.wikipedia.org/wiki/SpaceX'}
+
+```
+
+<sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyBashJSON></sub>
+
 #### Total number of archives

 ```bash
@ -326,15 +374,16 @@ https://web.archive.org/web/20120512142515/https://www.facebook.com/
 #### Get the source code

 ```bash
-waybackpy --url google.com --user_agent "my-unique-user-agent" --get url # Prints the source code of the url
+waybackpy --url google.com --user_agent "my-unique-user-agent" --get url # Prints the source code of the URL
 waybackpy --url google.com --user_agent "my-unique-user-agent" --get oldest # Prints the source code of the oldest archive
 waybackpy --url google.com --user_agent "my-unique-user-agent" --get newest # Prints the source code of the newest archive
-waybackpy --url google.com --user_agent "my-unique-user-agent" --get save # Save a new archive on wayback machine then print the source code of this archive.
+waybackpy --url google.com --user_agent "my-unique-user-agent" --get save # Save a new archive on Wayback machine then print the source code of this archive.
 ```

 <sub>Try this out in your browser @ <https://repl.it/@akamhy/WaybackPyBashGet></sub>

-####  Fetch all the URLs that the Wayback Machine knows for a domain
+#### Fetch all the URLs that the Wayback Machine knows for a domain
+
 1) You can add the '--alive' flag to only fetch alive links.
 2) You can add the '--subdomain' flag to add subdomains.
 3) '--alive' and '--subdomain' flags can be used simultaneously.
@ -345,19 +394,19 @@ pip install waybackpy

 # Ignore the above installation line.

-waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls 
+waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls
 # Prints all known URLs under akamhy.github.io


-waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --alive 
+waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --alive
 # Prints all known URLs under akamhy.github.io which are still working and not dead links.


-waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain 
-# Prints all known URLs under akamhy.github.io inclusing subdomain
+waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain
+# Prints all known URLs under akamhy.github.io including subdomain


-waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain --alive 
+waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain --alive
 # Prints all known URLs under akamhy.github.io including subdomain which are not dead links and still alive.

 ```
@ -366,11 +415,28 @@ waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --sub

 ## Tests

-[Here](https://github.com/akamhy/waybackpy/tree/master/tests)
+To run tests locally:

-## Dependency
+1) Install or update the testing/coverage tools 
+
+```bash
+pip install codecov pytest pytest-cov -U
+```
+
+2) Inside the repository run the following commands
+
+```bash
+pytest --cov=waybackpy tests/
+```
+
+3) To report coverage run
+
+```bash
+bash <(curl -s https://codecov.io/bash) -t SECRET_CODECOV_TOKEN
+```
+
+You can find the tests [here](https://github.com/akamhy/waybackpy/tree/master/tests).

-None, just python standard libraries (re, json, urllib, argparse and datetime). Both python 2 and 3 are supported :)

 ## Packaging

--- a/_config.yml
+++ b/_config.yml
@ -1 +1 @@
-theme: jekyll-theme-cayman
+theme: jekyll-theme-cayman
--- a/assets/waybackpy
+++ b/assets/waybackpy
--- a/assets/waybackpy-colored
+++ b/assets/waybackpy-colored
--- a/assets/waybackpy_logo.svg
+++ b/assets/waybackpy_logo.svg
@ -0,0 +1,85 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   id="svg8"
+   version="1.1"
+   viewBox="0 0 176.61171 41.907883"
+   height="41.907883mm"
+   width="176.61171mm">
+  <defs
+     id="defs2" />
+  <metadata
+     id="metadata5">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     transform="translate(-0.74835286,-98.31182)"
+     id="layer1">
+    <flowRoot
+       transform="scale(0.26458333)"
+       style="font-style:normal;font-weight:normal;font-size:40px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none"
+       id="flowRoot4598"
+       xml:space="preserve"><flowRegion
+         id="flowRegion4600"><rect
+           y="415.4129"
+           x="-38.183765"
+           height="48.08326"
+           width="257.38687"
+           id="rect4602" /></flowRegion><flowPara
+         id="flowPara4604"></flowPara></flowRoot>    <text
+       transform="scale(0.86288797,1.158899)"
+       id="text4777"
+       y="110.93711"
+       x="0.93061"
+       style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:28.14887619px;line-height:4.25;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Bold';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#003dff;fill-opacity:1;stroke:none;stroke-width:7.51955223;stroke-miterlimit:4;stroke-dasharray:none"
+       xml:space="preserve"><tspan
+         style="stroke-width:7.51955223"
+         id="tspan4775"
+         y="110.93711"
+         x="0.93061"><tspan
+           id="tspan4773"
+           style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:28.14887619px;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Bold';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:3.56786728px;writing-mode:lr-tb;text-anchor:start;fill:#003dff;fill-opacity:1;stroke-width:7.51955223;stroke-miterlimit:4;stroke-dasharray:none"
+           y="110.93711"
+           x="0.93061">waybackpy</tspan></tspan></text>
+    <rect
+       y="98.311821"
+       x="1.4967092"
+       height="4.8643045"
+       width="153.78688"
+       id="rect4644"
+       style="opacity:1;fill:#000080;fill-opacity:1;stroke:#00ff00;stroke-width:0;stroke-miterlimit:4;stroke-dasharray:none" />
+    <rect
+       style="opacity:1;fill:#000080;fill-opacity:1;stroke:#00ff00;stroke-width:0;stroke-miterlimit:4;stroke-dasharray:none"
+       id="rect4648"
+       width="153.78688"
+       height="4.490128"
+       x="23.573174"
+       y="135.72957" />
+    <rect
+       y="135.72957"
+       x="0.74835336"
+       height="4.4901319"
+       width="22.82482"
+       id="rect4650"
+       style="opacity:1;fill:#ff00ff;fill-opacity:1;stroke:#00ff00;stroke-width:0;stroke-miterlimit:4;stroke-dasharray:none" />
+    <rect
+       style="opacity:1;fill:#ff00ff;fill-opacity:1;stroke:#00ff00;stroke-width:0;stroke-miterlimit:4;stroke-dasharray:none"
+       id="rect4652"
+       width="21.702286"
+       height="4.8643003"
+       x="155.2836"
+       y="98.311821" />
+  </g>
+</svg>
--- a/index.rst
+++ b/index.rst
@ -1,386 +0,0 @@
-waybackpy
-=========
-
-|contributions welcome| |Build Status| |codecov| |Downloads| |Release|
-|Codacy Badge| |Maintainability| |CodeFactor| |made-with-python| |pypi|
-|PyPI - Python Version| |Maintenance| |Repo size| |License: MIT|
-
-|Internet Archive| |Wayback Machine|
-
-Waybackpy is a Python library that interfaces with the `Internet
-Archive <https://en.wikipedia.org/wiki/Internet_Archive>`__'s `Wayback
-Machine <https://en.wikipedia.org/wiki/Wayback_Machine>`__ API. Archive
-pages and retrieve archived pages easily.
-
-Table of contents
-=================
-
-.. raw:: html
-
-   <!--ts-->
-
-  `Installation <#installation>`__
-
-  `Usage <#usage>`__
-  `As a Python package <#as-a-python-package>`__
-
-   -  `Saving an url using
-      save() <#capturing-aka-saving-an-url-using-save>`__
-   -  `Receiving the oldest archive for an URL Using
-      oldest() <#receiving-the-oldest-archive-for-an-url-using-oldest>`__
-   -  `Receiving the recent most/newest archive for an URL using
-      newest() <#receiving-the-newest-archive-for-an-url-using-newest>`__
-   -  `Receiving archive close to a specified year, month, day, hour,
-      and minute using
-      near() <#receiving-archive-close-to-a-specified-year-month-day-hour-and-minute-using-near>`__
-   -  `Get the content of webpage using
-      get() <#get-the-content-of-webpage-using-get>`__
-   -  `Count total archives for an URL using
-      total\_archives() <#count-total-archives-for-an-url-using-total_archives>`__
-
-  `With Command-line interface <#with-the-command-line-interface>`__
-
-   -  `Save <#save>`__
-   -  `Oldest archive <#oldest-archive>`__
-   -  `Newest archive <#newest-archive>`__
-   -  `Total archives <#total-number-of-archives>`__
-   -  `Archive near a time <#archive-near-time>`__
-   -  `Get the source code <#get-the-source-code>`__
-
-  `Tests <#tests>`__
-
-  `Dependency <#dependency>`__
-
-  `License <#license>`__
-
-.. raw:: html
-
-   <!--te-->
-
-Installation
------------
-
-Using `pip <https://en.wikipedia.org/wiki/Pip_(package_manager)>`__:
-
-.. code:: bash
-
-    pip install waybackpy
-
-or direct from this repository using git.
-
-.. code:: bash
-
-    pip install git+https://github.com/akamhy/waybackpy.git
-
-Usage
-----
-
-As a Python package
-~~~~~~~~~~~~~~~~~~~
-
-Capturing aka Saving an url using save()
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. code:: python
-
-    import waybackpy
-
-    new_archive_url = waybackpy.Url(
-
-        url = "https://en.wikipedia.org/wiki/Multivariable_calculus",
-        user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"
-        
-    ).save()
-
-    print(new_archive_url)
-
-.. code:: bash
-
-    https://web.archive.org/web/20200504141153/https://github.com/akamhy/waybackpy
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPySaveExample\ 
-
-Receiving the oldest archive for an URL using oldest()
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. code:: python
-
-    import waybackpy
-
-    oldest_archive_url = waybackpy.Url(
-
-        "https://www.google.com/",
-        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:40.0) Gecko/20100101 Firefox/40.0"
-        
-    ).oldest()
-
-    print(oldest_archive_url)
-
-.. code:: bash
-
-    http://web.archive.org/web/19981111184551/http://google.com:80/
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPyOldestExample\ 
-
-Receiving the newest archive for an URL using newest()
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. code:: python
-
-    import waybackpy
-
-    newest_archive_url = waybackpy.Url(
-
-        "https://www.facebook.com/",
-        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
-        
-    ).newest()
-
-    print(newest_archive_url)
-
-.. code:: bash
-
-    https://web.archive.org/web/20200714013225/https://www.facebook.com/
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPyNewestExample\ 
-
-Receiving archive close to a specified year, month, day, hour, and minute using near()
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. code:: python
-
-    from waybackpy import Url
-
-    user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Firefox/38.0"
-    github_url = "https://github.com/"
-
-
-    github_wayback_obj = Url(github_url, user_agent)
-
-    # Do not pad (don't use zeros in the month, year, day, minute, and hour arguments). e.g. For January, set month = 1 and not month = 01.
-
-.. code:: python
-
-    github_archive_near_2010 = github_wayback_obj.near(year=2010)
-    print(github_archive_near_2010)
-
-.. code:: bash
-
-    https://web.archive.org/web/20100719134402/http://github.com/
-
-.. code:: python
-
-    github_archive_near_2011_may = github_wayback_obj.near(year=2011, month=5)
-    print(github_archive_near_2011_may)
-
-.. code:: bash
-
-    https://web.archive.org/web/20110519185447/https://github.com/
-
-.. code:: python
-
-    github_archive_near_2015_january_26 = github_wayback_obj.near(
-        year=2015, month=1, day=26
-    )
-    print(github_archive_near_2015_january_26)
-
-.. code:: bash
-
-    https://web.archive.org/web/20150127031159/https://github.com
-
-.. code:: python
-
-    github_archive_near_2018_4_july_9_2_am = github_wayback_obj.near(
-        year=2018, month=7, day=4, hour = 9, minute = 2
-    )
-    print(github_archive_near_2018_4_july_9_2_am)
-
-.. code:: bash
-
-    https://web.archive.org/web/20180704090245/https://github.com/
-
-The library doesn't supports seconds yet. You are encourged to create a
-PR ;)
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPyNearExample\ 
-
-Get the content of webpage using get()
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. code:: python
-
-    import waybackpy
-
-    google_url = "https://www.google.com/"
-
-    User_Agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36"
-
-    waybackpy_url_object = waybackpy.Url(google_url, User_Agent)
-
-
-    # If no argument is passed in get(), it gets the source of the Url used to create the object.
-    current_google_url_source = waybackpy_url_object.get()
-    print(current_google_url_source)
-
-
-    # The following chunk of code will force a new archive of google.com and get the source of the archived page.
-    # waybackpy_url_object.save() type is string.
-    google_newest_archive_source = waybackpy_url_object.get(
-        waybackpy_url_object.save()
-    )
-    print(google_newest_archive_source)
-
-
-    # waybackpy_url_object.oldest() type is str, it's oldest archive of google.com
-    google_oldest_archive_source = waybackpy_url_object.get(
-        waybackpy_url_object.oldest()
-    )
-    print(google_oldest_archive_source)
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPyGetExample#main.py\ 
-
-Count total archives for an URL using total\_archives()
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. code:: python
-
-    import waybackpy
-
-    URL = "https://en.wikipedia.org/wiki/Python (programming language)"
-
-    UA = "Mozilla/5.0 (iPad; CPU OS 8_1_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B435 Safari/600.1.4"
-
-    archive_count = waybackpy.Url(
-        url=URL,
-        user_agent=UA
-    ).total_archives()
-
-    print(archive_count) # total_archives() returns an int
-
-.. code:: bash
-
-    2440
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPyTotalArchivesExample\ 
-
-With the Command-line interface
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Save
-^^^^
-
-.. code:: bash
-
-    $ waybackpy --url "https://en.wikipedia.org/wiki/Social_media" --user_agent "my-unique-user-agent" --save
-    https://web.archive.org/web/20200719062108/https://en.wikipedia.org/wiki/Social_media
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPyBashSave\ 
-
-Oldest archive
-^^^^^^^^^^^^^^
-
-.. code:: bash
-
-    $ waybackpy --url "https://en.wikipedia.org/wiki/SpaceX" --user_agent "my-unique-user-agent" --oldest
-    https://web.archive.org/web/20040803000845/http://en.wikipedia.org:80/wiki/SpaceX
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPyBashOldest\ 
-
-Newest archive
-^^^^^^^^^^^^^^
-
-.. code:: bash
-
-    $ waybackpy --url "https://en.wikipedia.org/wiki/YouTube" --user_agent "my-unique-user-agent" --newest
-    https://web.archive.org/web/20200606044708/https://en.wikipedia.org/wiki/YouTube
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPyBashNewest\ 
-
-Total number of archives
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. code:: bash
-
-    $ waybackpy --url "https://en.wikipedia.org/wiki/Linux_kernel" --user_agent "my-unique-user-agent" --total
-    853
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPyBashTotal\ 
-
-Archive near time
-^^^^^^^^^^^^^^^^^
-
-.. code:: bash
-
-    $ waybackpy --url facebook.com --user_agent "my-unique-user-agent" --near --year 2012 --month 5 --day 12
-    https://web.archive.org/web/20120512142515/https://www.facebook.com/
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPyBashNear\ 
-
-Get the source code
-^^^^^^^^^^^^^^^^^^^
-
-.. code:: bash
-
-    $ waybackpy --url google.com --user_agent "my-unique-user-agent" --get url # Prints the source code of the url
-    $ waybackpy --url google.com --user_agent "my-unique-user-agent" --get oldest # Prints the source code of the oldest archive
-    $ waybackpy --url google.com --user_agent "my-unique-user-agent" --get newest # Prints the source code of the newest archive
-    $ waybackpy --url google.com --user_agent "my-unique-user-agent" --get save # Save a new archive on wayback machine then print the source code of this archive.
-
-Try this out in your browser @
-https://repl.it/@akamhy/WaybackPyBashGet\ 
-
-Tests
-----
-
-  `Here <https://github.com/akamhy/waybackpy/tree/master/tests>`__
-
-Dependency
----------
-
-  None, just python standard libraries (re, json, urllib, argparse and
-   datetime). Both python 2 and 3 are supported :)
-
-License
-------
-
-`MIT
-License <https://github.com/akamhy/waybackpy/blob/master/LICENSE>`__
-
-.. |contributions welcome| image:: https://img.shields.io/static/v1.svg?label=Contributions&message=Welcome&color=0059b3&style=flat-square
-.. |Build Status| image:: https://img.shields.io/travis/akamhy/waybackpy.svg?label=Travis%20CI&logo=travis&style=flat-square
-   :target: https://travis-ci.org/akamhy/waybackpy
-.. |codecov| image:: https://codecov.io/gh/akamhy/waybackpy/branch/master/graph/badge.svg
-   :target: https://codecov.io/gh/akamhy/waybackpy
-.. |Downloads| image:: https://pepy.tech/badge/waybackpy/month
-   :target: https://pepy.tech/project/waybackpy/month
-.. |Release| image:: https://img.shields.io/github/v/release/akamhy/waybackpy.svg
-   :target: https://github.com/akamhy/waybackpy/releases
-.. |Codacy Badge| image:: https://api.codacy.com/project/badge/Grade/255459cede9341e39436ec8866d3fb65
-   :target: https://www.codacy.com/manual/akamhy/waybackpy?utm_source=github.com&utm_medium=referral&utm_content=akamhy/waybackpy&utm_campaign=Badge_Grade
-.. |Maintainability| image:: https://api.codeclimate.com/v1/badges/942f13d8177a56c1c906/maintainability
-   :target: https://codeclimate.com/github/akamhy/waybackpy/maintainability
-.. |CodeFactor| image:: https://www.codefactor.io/repository/github/akamhy/waybackpy/badge
-   :target: https://www.codefactor.io/repository/github/akamhy/waybackpy
-.. |made-with-python| image:: https://img.shields.io/badge/Made%20with-Python-1f425f.svg
-   :target: https://www.python.org/
-.. |pypi| image:: https://img.shields.io/pypi/v/waybackpy.svg
-   :target: https://pypi.org/project/waybackpy/
-.. |PyPI - Python Version| image:: https://img.shields.io/pypi/pyversions/waybackpy?style=flat-square
-.. |Maintenance| image:: https://img.shields.io/badge/Maintained%3F-yes-green.svg
-   :target: https://github.com/akamhy/waybackpy/graphs/commit-activity
-.. |Repo size| image:: https://img.shields.io/github/repo-size/akamhy/waybackpy.svg?label=Repo%20size&style=flat-square
-.. |License: MIT| image:: https://img.shields.io/badge/License-MIT-yellow.svg
-   :target: https://github.com/akamhy/waybackpy/blob/master/LICENSE
-.. |Internet Archive| image:: https://upload.wikimedia.org/wikipedia/commons/thumb/8/84/Internet_Archive_logo_and_wordmark.svg/84px-Internet_Archive_logo_and_wordmark.svg.png
-.. |Wayback Machine| image:: https://upload.wikimedia.org/wikipedia/commons/thumb/0/01/Wayback_Machine_logo_2010.svg/284px-Wayback_Machine_logo_2010.svg.png
--- a/requirements.txt
+++ b/requirements.txt
@ -0,0 +1 @@
+requests>=2.24.0
--- a/setup.py
+++ b/setup.py
@ -1,54 +1,54 @@
 import os.path
 from setuptools import setup

-with open(os.path.join(os.path.dirname(__file__), 'README.md')) as f:
+with open(os.path.join(os.path.dirname(__file__), "README.md")) as f:
    long_description = f.read()

 about = {}
-with open(os.path.join(os.path.dirname(__file__), 'waybackpy', '__version__.py')) as f:
+with open(os.path.join(os.path.dirname(__file__), "waybackpy", "__version__.py")) as f:
    exec(f.read(), about)
-    
+
 setup(
-    name = about['__title__'],
-    packages = ['waybackpy'],
-    version = about['__version__'],
-    description = about['__description__'],
+    name=about["__title__"],
+    packages=["waybackpy"],
+    version=about["__version__"],
+    description=about["__description__"],
    long_description=long_description,
-    long_description_content_type='text/markdown',
-    license= about['__license__'],
-    author = about['__author__'],
-    author_email = about['__author_email__'],
-    url = about['__url__'],
-    download_url = 'https://github.com/akamhy/waybackpy/archive/2.1.8.tar.gz',
-    keywords = ['wayback', 'archive', 'archive website', 'wayback machine', 'Internet Archive'],
-    install_requires=[],
-    python_requires= ">=2.7",
+    long_description_content_type="text/markdown",
+    license=about["__license__"],
+    author=about["__author__"],
+    author_email=about["__author_email__"],
+    url=about["__url__"],
+    download_url="https://github.com/akamhy/waybackpy/archive/2.3.2.tar.gz",
+    keywords=[
+        "Archive It",
+        "Archive Website",
+        "Wayback Machine",
+        "waybackurls",
+        "Internet Archive",
+    ],
+    install_requires=["requests"],
+    python_requires=">=3.4",
    classifiers=[
-        'Development Status :: 5 - Production/Stable',
-        'Intended Audience :: Developers',
-        'Natural Language :: English',
-        'Topic :: Software Development :: Build Tools',
-        'License :: OSI Approved :: MIT License',
-        'Programming Language :: Python',
-        'Programming Language :: Python :: 2',
-        'Programming Language :: Python :: 2.7',
-        'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.2',
-        'Programming Language :: Python :: 3.3',
-        'Programming Language :: Python :: 3.4',
-        'Programming Language :: Python :: 3.5',
-        'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3.7',
-        'Programming Language :: Python :: 3.8',
-        'Programming Language :: Python :: Implementation :: CPython',
-        ],
-    entry_points={
-        'console_scripts': [
-            'waybackpy = waybackpy.cli:main'
-        ]
-    },
+        "Development Status :: 5 - Production/Stable",
+        "Intended Audience :: Developers",
+        "Natural Language :: English",
+        "Topic :: Software Development :: Build Tools",
+        "License :: OSI Approved :: MIT License",
+        "Programming Language :: Python",
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.4",
+        "Programming Language :: Python :: 3.5",
+        "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
+        "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: Implementation :: CPython",
+    ],
+    entry_points={"console_scripts": ["waybackpy = waybackpy.cli:main"]},
    project_urls={
-        'Documentation': 'https://waybackpy.readthedocs.io',
-        'Source': 'https://github.com/akamhy/waybackpy',
+        "Documentation": "https://akamhy.github.io/waybackpy/",
+        "Source": "https://github.com/akamhy/waybackpy",
+        "Tracker": "https://github.com/akamhy/waybackpy/issues",
    },
 )
--- a/tests/init.py
+++ b/tests/init.py
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@ -6,82 +6,291 @@ import argparse

 sys.path.append("..")
 import waybackpy.cli as cli  # noqa: E402
-from waybackpy.wrapper import  Url  # noqa: E402
+from waybackpy.wrapper import Url  # noqa: E402
 from waybackpy.__version__ import __version__

-codecov_python = False
-if sys.version_info > (3, 7):
-    codecov_python = True
-
 # Namespace(day=None, get=None, hour=None, minute=None, month=None, near=False,
 # newest=False, oldest=False, save=False, total=False, url=None, user_agent=None, version=False, year=None)

-if codecov_python:
-    def test_save():
-        args = argparse.Namespace(user_agent=None, url="https://pypi.org/user/akamhy/", total=False, version=False,
-        oldest=False, save=True, newest=False, near=False, alive=False, subdomain=False, known_urls=False, get=None)
-        reply = cli.args_handler(args)
-        assert "pypi.org/user/akamhy" in reply
+
+def test_save():
+    args = argparse.Namespace(
+        user_agent=None,
+        url="https://pypi.org/user/akamhy/",
+        total=False,
+        version=False,
+        oldest=False,
+        save=True,
+        json=False,
+        archive_url=False,
+        newest=False,
+        near=False,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get=None,
+    )
+    reply = cli.args_handler(args)
+    assert "pypi.org/user/akamhy" in str(reply)
+
+
+def test_json():
+    args = argparse.Namespace(
+        user_agent=None,
+        url="https://pypi.org/user/akamhy/",
+        total=False,
+        version=False,
+        oldest=False,
+        save=False,
+        json=True,
+        archive_url=False,
+        newest=False,
+        near=False,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get=None,
+    )
+    reply = cli.args_handler(args)
+    assert "archived_snapshots" in str(reply)
+
+
+def test_archive_url():
+    args = argparse.Namespace(
+        user_agent=None,
+        url="https://pypi.org/user/akamhy/",
+        total=False,
+        version=False,
+        oldest=False,
+        save=False,
+        json=False,
+        archive_url=True,
+        newest=False,
+        near=False,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get=None,
+    )
+    reply = cli.args_handler(args)
+    assert "https://web.archive.org/web/" in str(reply)
+

 def test_oldest():
-    args = argparse.Namespace(user_agent=None, url="https://pypi.org/user/akamhy/", total=False, version=False,
-    oldest=True, save=False, newest=False, near=False, alive=False, subdomain=False, known_urls=False, get=None)
+    args = argparse.Namespace(
+        user_agent=None,
+        url="https://pypi.org/user/akamhy/",
+        total=False,
+        version=False,
+        oldest=True,
+        save=False,
+        json=False,
+        archive_url=False,
+        newest=False,
+        near=False,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get=None,
+    )
    reply = cli.args_handler(args)
-    assert "pypi.org/user/akamhy" in reply
+    assert "pypi.org/user/akamhy" in str(reply)
+

 def test_newest():
-    args = argparse.Namespace(user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
-    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9", url="https://pypi.org/user/akamhy/", total=False, version=False,
-    oldest=False, save=False, newest=True, near=False, alive=False, subdomain=False, known_urls=False, get=None)
+    args = argparse.Namespace(
+        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
+    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9",
+        url="https://pypi.org/user/akamhy/",
+        total=False,
+        version=False,
+        oldest=False,
+        save=False,
+        json=False,
+        archive_url=False,
+        newest=True,
+        near=False,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get=None,
+    )
    reply = cli.args_handler(args)
-    assert "pypi.org/user/akamhy" in reply
+    assert "pypi.org/user/akamhy" in str(reply)
+

 def test_total_archives():
-    args = argparse.Namespace(user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
-    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9", url="https://pypi.org/user/akamhy/", total=True, version=False,
-    oldest=False, save=False, newest=False, near=False, alive=False, subdomain=False, known_urls=False, get=None)
+    args = argparse.Namespace(
+        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
+    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9",
+        url="https://pypi.org/user/akamhy/",
+        total=True,
+        version=False,
+        oldest=False,
+        save=False,
+        json=False,
+        archive_url=False,
+        newest=False,
+        near=False,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get=None,
+    )
    reply = cli.args_handler(args)
    assert isinstance(reply, int)

-def test_near():
-    args = argparse.Namespace(user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
-    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9", url="https://pypi.org/user/akamhy/", total=False, version=False,
-    oldest=False, save=False, newest=False, near=True, alive=False, subdomain=False, known_urls=False, get=None, year=2020, month=7, day=15, hour=1, minute=1)
+
+def test_known_urls():
+    args = argparse.Namespace(
+        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
+    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9",
+        url="https://akamhy.github.io",
+        total=False,
+        version=False,
+        oldest=False,
+        save=False,
+        json=False,
+        archive_url=False,
+        newest=False,
+        near=False,
+        alive=True,
+        subdomain=True,
+        known_urls=True,
+        get=None,
+    )
    reply = cli.args_handler(args)
-    assert "202007" in reply
+    assert "github" in str(reply)
+
+
+def test_near():
+    args = argparse.Namespace(
+        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
+    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9",
+        url="https://pypi.org/user/akamhy/",
+        total=False,
+        version=False,
+        oldest=False,
+        save=False,
+        json=False,
+        archive_url=False,
+        newest=False,
+        near=True,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get=None,
+        year=2020,
+        month=7,
+        day=15,
+        hour=1,
+        minute=1,
+    )
+    reply = cli.args_handler(args)
+    assert "202007" in str(reply)


 def test_get():
-    args = argparse.Namespace(user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
-    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9", url="https://pypi.org/user/akamhy/", total=False, version=False,
-    oldest=False, save=False, newest=False, near=False, alive=False, subdomain=False, known_urls=False, get="url")
+    args = argparse.Namespace(
+        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
+    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9",
+        url="https://pypi.org/user/akamhy/",
+        total=False,
+        version=False,
+        oldest=False,
+        save=False,
+        json=False,
+        archive_url=False,
+        newest=False,
+        near=False,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get="url",
+    )
    reply = cli.args_handler(args)
-    assert "waybackpy" in reply
+    assert "waybackpy" in str(reply)

-    args = argparse.Namespace(user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
-    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9", url="https://pypi.org/user/akamhy/", total=False, version=False,
-    oldest=False, save=False, newest=False, near=False, alive=False, subdomain=False, known_urls=False, get="oldest")
+    args = argparse.Namespace(
+        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
+    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9",
+        url="https://pypi.org/user/akamhy/",
+        total=False,
+        version=False,
+        oldest=False,
+        save=False,
+        json=False,
+        archive_url=False,
+        newest=False,
+        near=False,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get="oldest",
+    )
    reply = cli.args_handler(args)
-    assert "waybackpy" in reply
+    assert "waybackpy" in str(reply)

-    args = argparse.Namespace(user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
-    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9", url="https://pypi.org/user/akamhy/", total=False, version=False,
-    oldest=False, save=False, newest=False, near=False, alive=False, subdomain=False, known_urls=False, get="newest")
+    args = argparse.Namespace(
+        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
+    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9",
+        url="https://pypi.org/user/akamhy/",
+        total=False,
+        version=False,
+        oldest=False,
+        save=False,
+        json=False,
+        archive_url=False,
+        newest=False,
+        near=False,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get="newest",
+    )
    reply = cli.args_handler(args)
-    assert "waybackpy" in reply
+    assert "waybackpy" in str(reply)

-    if codecov_python:
-        args = argparse.Namespace(user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
-        (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9", url="https://pypi.org/user/akamhy/", total=False, version=False,
-        oldest=False, save=False, newest=False, near=False, alive=False, subdomain=False, known_urls=False, get="save")
-        reply = cli.args_handler(args)
-        assert "waybackpy" in reply
-
-    args = argparse.Namespace(user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
-    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9", url="https://pypi.org/user/akamhy/", total=False, version=False,
-    oldest=False, save=False, newest=False, near=False, alive=False, subdomain=False, known_urls=False, get="BullShit")
+    args = argparse.Namespace(
+        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
+    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9",
+        url="https://pypi.org/user/akamhy/",
+        total=False,
+        version=False,
+        oldest=False,
+        save=False,
+        json=False,
+        archive_url=False,
+        newest=False,
+        near=False,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get="save",
+    )
    reply = cli.args_handler(args)
-    assert "get the source code of the" in reply
+    assert "waybackpy" in str(reply)
+
+    args = argparse.Namespace(
+        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 \
+    (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9",
+        url="https://pypi.org/user/akamhy/",
+        total=False,
+        version=False,
+        oldest=False,
+        save=False,
+        json=False,
+        archive_url=False,
+        newest=False,
+        near=False,
+        alive=False,
+        subdomain=False,
+        known_urls=False,
+        get="BullShit",
+    )
+    reply = cli.args_handler(args)
+    assert "get the source code of the" in str(reply)
+

 def test_args_handler():
    args = argparse.Namespace(version=True)
@ -90,8 +299,9 @@ def test_args_handler():

    args = argparse.Namespace(url=None, version=False)
    reply = cli.args_handler(args)
-    assert ("waybackpy %s" % (__version__)) in reply
+    assert ("waybackpy %s" % (__version__)) in str(reply)
+

 def test_main():
    # This also tests the parse_args method in cli.py
-    cli.main(['temp.py', '--version'])
+    cli.main(["temp.py", "--version"])
--- a/tests/test_wrapper.py
+++ b/tests/test_wrapper.py
@ -2,16 +2,12 @@
 import sys
 import pytest
 import random
-import time
+import requests

 sys.path.append("..")
+
 import waybackpy.wrapper as waybackpy  # noqa: E402

-if sys.version_info >= (3, 0):  # If the python ver >= 3
-    from urllib.request import Request, urlopen
-    from urllib.error import URLError
-else:  # For python2.x
-    from urllib2 import Request, urlopen, URLError

 user_agent = "Mozilla/5.0 (Windows NT 6.2; rv:20.0) Gecko/20121202 Firefox/20.0"

@ -23,22 +19,25 @@ def test_clean_url():
    test_result = target._clean_url()
    assert answer == test_result

+
 def test_dunders():
    url = "https://en.wikipedia.org/wiki/Network_security"
    user_agent = "UA"
    target = waybackpy.Url(url, user_agent)
    assert "waybackpy.Url(url=%s, user_agent=%s)" % (url, user_agent) == repr(target)
-    assert len(target) == len(url)
-    assert str(target) == url
+    assert "en.wikipedia.org" in str(target)
+

 def test_archive_url_parser():
-    request_url = "https://amazon.com"
-    hdr = {"User-Agent": user_agent}  # nosec
-    req = Request(request_url, headers=hdr)  # nosec
-    header = waybackpy._get_response(req).headers
+    endpoint = "https://amazon.com"
+    user_agent = "Mozilla/5.0 (Windows NT 6.2; rv:20.0) Gecko/20121202 Firefox/20.0"
+    headers = {"User-Agent": "%s" % user_agent}
+    response = waybackpy._get_response(endpoint, params=None, headers=headers)
+    header = response.headers
    with pytest.raises(Exception):
        waybackpy._archive_url_parser(header)

+
 def test_url_check():
    broken_url = "http://wwwgooglecom/"
    with pytest.raises(Exception):
@ -47,7 +46,6 @@ def test_url_check():

 def test_save():
    # Test for urls that exist and can be archived.
-    time.sleep(10)

    url_list = [
        "en.wikipedia.org",
@ -64,43 +62,26 @@ def test_save():
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36",
    )
-    archived_url1 = target.save()
+    archived_url1 = str(target.save())
    assert url1 in archived_url1

-    if sys.version_info > (3, 6):
+    # Test for urls that are incorrect.
+    with pytest.raises(Exception):
+        url2 = "ha ha ha ha"
+        waybackpy.Url(url2, user_agent)
+    url3 = "http://www.archive.is/faq.html"

-        # Test for urls that are incorrect.
-        with pytest.raises(Exception):
-            url2 = "ha ha ha ha"
-            waybackpy.Url(url2, user_agent)
-        time.sleep(5)
-        url3 = "http://www.archive.is/faq.html"
-        # Test for urls not allowed to archive by robot.txt. Doesn't works anymore. Find alternatives.
-#         with pytest.raises(Exception):
-#             
-#             target = waybackpy.Url(
-#                 url3,
-#                 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:25.0) "
-#                 "Gecko/20100101 Firefox/25.0",
-#             )
-#             target.save()
-#         time.sleep(5)
-        # Non existent urls, test
-        with pytest.raises(Exception):
-            target = waybackpy.Url(
-                url3,
-                "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) "
-                "AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 "
-                "Safari/533.20.27",
-            )
-            target.save()
-
-    else:
-        pass
+    with pytest.raises(Exception):
+        target = waybackpy.Url(
+            url3,
+            "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) "
+            "AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 "
+            "Safari/533.20.27",
+        )
+        target.save()


 def test_near():
-    time.sleep(10)
    url = "google.com"
    target = waybackpy.Url(
        url,
@ -108,51 +89,59 @@ def test_near():
        "(KHTML, like Gecko) Version/5.0.3 Safari/533.19.4",
    )
    archive_near_year = target.near(year=2010)
-    assert "2010" in archive_near_year
+    assert "2010" in str(archive_near_year)

-    if sys.version_info > (3, 6):
-        time.sleep(5)
-        archive_near_month_year = target.near(year=2015, month=2)
-        assert (
-            ("201502" in archive_near_month_year)
-            or ("201501" in archive_near_month_year)
-            or ("201503" in archive_near_month_year)
-        )
+    archive_near_month_year = str(target.near(year=2015, month=2))
+    assert (
+        ("201502" in archive_near_month_year)
+        or ("201501" in archive_near_month_year)
+        or ("201503" in archive_near_month_year)
+    )

-        target = waybackpy.Url(
-            "www.python.org",
-            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
-            "(KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246",
-        )
-        archive_near_hour_day_month_year = target.near(
-            year=2008, month=5, day=9, hour=15
-        )
-        assert (
-            ("2008050915" in archive_near_hour_day_month_year)
-            or ("2008050914" in archive_near_hour_day_month_year)
-            or ("2008050913" in archive_near_hour_day_month_year)
-        )
+    target = waybackpy.Url(
+        "www.python.org",
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
+        "(KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246",
+    )
+    archive_near_hour_day_month_year = str(
+        target.near(year=2008, month=5, day=9, hour=15)
+    )
+    assert (
+        ("2008050915" in archive_near_hour_day_month_year)
+        or ("2008050914" in archive_near_hour_day_month_year)
+        or ("2008050913" in archive_near_hour_day_month_year)
+    )

-        with pytest.raises(Exception):
-            NeverArchivedUrl = (
-                "https://ee_3n.wrihkeipef4edia.org/rwti5r_ki/Nertr6w_rork_rse7c_urity"
-            )
-            target = waybackpy.Url(NeverArchivedUrl, user_agent)
-            target.near(year=2010)
-    else:
-        pass
+    with pytest.raises(Exception):
+        NeverArchivedUrl = (
+            "https://ee_3n.wrihkeipef4edia.org/rwti5r_ki/Nertr6w_rork_rse7c_urity"
+        )
+        target = waybackpy.Url(NeverArchivedUrl, user_agent)
+        target.near(year=2010)


 def test_oldest():
    url = "github.com/akamhy/waybackpy"
    target = waybackpy.Url(url, user_agent)
-    assert "20200504141153" in target.oldest()
+    assert "20200504141153" in str(target.oldest())
+
+
+def test_json():
+    url = "github.com/akamhy/waybackpy"
+    target = waybackpy.Url(url, user_agent)
+    assert "archived_snapshots" in str(target.JSON)
+
+
+def test_archive_url():
+    url = "github.com/akamhy/waybackpy"
+    target = waybackpy.Url(url, user_agent)
+    assert "github.com/akamhy" in str(target.archive_url)


 def test_newest():
    url = "github.com/akamhy/waybackpy"
    target = waybackpy.Url(url, user_agent)
-    assert url in target.newest()
+    assert url in str(target.newest())


 def test_get():
@ -160,31 +149,36 @@ def test_get():
    assert "Welcome to Google" in target.get(target.oldest())


-
 def test_wayback_timestamp():
-    ts = waybackpy._wayback_timestamp(
-        year=2020, month=1, day=2, hour=3, minute=4
-    )
+    ts = waybackpy._wayback_timestamp(year=2020, month=1, day=2, hour=3, minute=4)
    assert "202001020304" in str(ts)


 def test_get_response():
-    hdr = {
-        "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) "
-        "Gecko/20100101 Firefox/78.0"
-    }
-    req = Request("https://www.google.com", headers=hdr)  # nosec
-    response = waybackpy._get_response(req)
-    assert response.code == 200
+    endpoint = "https://www.google.com"
+    user_agent = (
+        "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
+    )
+    headers = {"User-Agent": "%s" % user_agent}
+    response = waybackpy._get_response(endpoint, params=None, headers=headers)
+    assert response.status_code == 200


 def test_total_archives():
-    if sys.version_info > (3, 6):
-        target = waybackpy.Url(" https://google.com ", user_agent)
-        assert target.total_archives() > 500000
-    else:
-        pass
+
+    target = waybackpy.Url(" https://google.com ", user_agent)
+    assert target.total_archives() > 500000
+
    target = waybackpy.Url(
        " https://gaha.e4i3n.m5iai3kip6ied.cima/gahh2718gs/ahkst63t7gad8 ", user_agent
    )
    assert target.total_archives() == 0
+
+
+def test_known_urls():
+
+    target = waybackpy.Url("akamhy.github.io", user_agent)
+    assert len(target.known_urls(alive=True, subdomain=True)) > 2
+
+    target = waybackpy.Url("akamhy.github.io", user_agent)
+    assert len(target.known_urls()) > 3
--- a/waybackpy/init.py
+++ b/waybackpy/init.py
@ -10,7 +10,7 @@
 # ━━━━━━━━━━━┗━━┛━━━━━━━━━━━━━━━━━━━━━━━━┗━━┛━

 """
-Waybackpy is a Python library that interfaces with the Internet Archive's Wayback Machine API.
+Waybackpy is a Python package that interfaces with the Internet Archive's Wayback Machine API.
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 Archive pages and retrieve archived pages easily.
--- a/waybackpy/version.py
+++ b/waybackpy/version.py
@ -1,10 +1,13 @@
 # -*- coding: utf-8 -*-

 __title__ = "waybackpy"
-__description__ = "A Python library that interfaces with the Internet Archive's Wayback Machine API. Archive pages and retrieve archived pages easily."
+__description__ = (
+    "A Python package that interfaces with the Internet Archive's Wayback Machine API. "
+    "Archive pages and retrieve archived pages easily."
+)
 __url__ = "https://akamhy.github.io/waybackpy/"
-__version__ = "2.1.8"
+__version__ = "2.3.2"
 __author__ = "akamhy"
-__author_email__ = "akash3pro@gmail.com"
+__author_email__ = "akamhy@yahoo.com"
 __license__ = "MIT"
-__copyright__ = "Copyright 2020 akamhy"
+__copyright__ = "Copyright 2020-2021 Akash Mahanty et al."
--- a/waybackpy/cli.py
+++ b/waybackpy/cli.py
@ -1,23 +1,76 @@
 # -*- coding: utf-8 -*-
-from __future__ import print_function
 import sys
 import os
 import re
 import argparse
+import string
+import random
 from waybackpy.wrapper import Url
 from waybackpy.__version__ import __version__
+from waybackpy.exceptions import WaybackError
+

 def _save(obj):
-    return (obj.save())
+    try:
+        return obj.save()
+    except Exception as err:
+        e = str(err)
+        url = obj.url
+        m = re.search(r"Header:\n(.*)", e)
+        if m:
+            header = m.group(1)
+        if "No archive URL found in the API response" in e:
+            return (
+                "\n[waybackpy] Can not save/archive your link.\n[waybackpy] This\
+                 could happen because either your waybackpy (%s) is likely out of\
+                 date or Wayback Machine is malfunctioning.\n[waybackpy] Visit\
+                 https://github.com/akamhy/waybackpy for the latest version of \
+                waybackpy.\n[waybackpy] API response Header :\n%s"
+                % (__version__, header)
+            )
+        return WaybackError(err)
+
+
+def _archive_url(obj):
+    return obj.archive_url
+
+
+def _json(obj):
+    return obj.JSON
+
+
+def handle_not_archived_error(e, obj):
+    m = re.search(r"archive\sfor\s\'(.*?)\'\stry", str(e))
+    if m:
+        url = m.group(1)
+        ua = obj.user_agent
+        if "github.com/akamhy/waybackpy" in ua:
+            ua = "YOUR_USER_AGENT_HERE"
+        return (
+            "\n[Waybackpy] Can not find archive for '%s'.\n[Waybackpy] You can"
+            " save the URL using the following command:\n[Waybackpy] waybackpy --"
+            'user_agent "%s" --url "%s" --save' % (url, ua, url)
+        )
+    return WaybackError(e)
+

 def _oldest(obj):
-    return (obj.oldest())
+    try:
+        return obj.oldest()
+    except Exception as e:
+        return handle_not_archived_error(e, obj)
+

 def _newest(obj):
-    return (obj.newest())
+    try:
+        return obj.newest()
+    except Exception as e:
+        return handle_not_archived_error(e, obj)
+

 def _total_archives(obj):
-    return (obj.total_archives())
+    return obj.total_archives()
+

 def _near(obj, args):
    _near_args = {}
@ -31,9 +84,37 @@ def _near(obj, args):
        _near_args["hour"] = args.hour
    if args.minute:
        _near_args["minute"] = args.minute
-    return (obj.near(**_near_args))
+
+    try:
+        return obj.near(**_near_args)
+    except Exception as e:
+        return handle_not_archived_error(e, obj)
+
+
+def _save_urls_on_file(input_list, live_url_count):
+    m = re.search("https?://([A-Za-z_0-9.-]+).*", input_list[0])
+    if m:
+        domain = m.group(1)
+    else:
+        domain = "domain-unknown"
+
+    uid = "".join(
+        random.choice(string.ascii_lowercase + string.digits) for _ in range(6)
+    )
+
+    file_name = "%s-%d-urls-%s.txt" % (domain, live_url_count, uid)
+    file_content = "\n".join(input_list)
+    file_path = os.path.join(os.getcwd(), file_name)
+    with open(file_path, "w+") as f:
+        f.write(file_content)
+    return "%s\n\n'%s' saved in current working directory" % (file_content, file_name)
+

 def _known_urls(obj, args):
+    """Abbreviations:
+    sd = subdomain
+    al = alive
+    """
    sd = False
    al = False
    if args.subdomain:
@ -44,116 +125,174 @@ def _known_urls(obj, args):
    total_urls = len(url_list)

    if total_urls > 0:
-        m = re.search('https?://([A-Za-z_0-9.-]+).*', url_list[0])
-        if m:
-            domain = m.group(1)
-        else:
-            domain = "waybackpy-known"
-        dir_path = os.path.abspath(os.getcwd())
-        file_name = dir_path + "/%s-%d-urls.txt" % (domain, total_urls)
-        text = "\n".join(url_list) + "\n"
-        with open(file_name, "a+") as f:
-            f.write(text)
-        text =  text + "%d URLs found and saved in ./%s-%d-urls.txt" % (
-            total_urls, domain, total_urls
-            )
-
+        text = _save_urls_on_file(url_list, total_urls)
    else:
        text = "No known URLs found. Please try a diffrent domain!"

    return text

+
 def _get(obj, args):
    if args.get.lower() == "url":
-        return (obj.get())
+        output = obj.get()

-    if args.get.lower() == "oldest":
-        return (obj.get(obj.oldest()))
+    elif args.get.lower() == "archive_url":
+        output = obj.get(obj.archive_url)

-    if args.get.lower() == "latest" or args.get.lower() == "newest":
-        return (obj.get(obj.newest()))
+    elif args.get.lower() == "oldest":
+        output = obj.get(obj.oldest())

-    if args.get.lower() == "save":
-        return (obj.get(obj.save()))
+    elif args.get.lower() == "latest" or args.get.lower() == "newest":
+        output = obj.get(obj.newest())
+
+    elif args.get.lower() == "save":
+        output = obj.get(obj.save())
+
+    else:
+        output = "Use get as \"--get 'source'\", 'source' can be one of the followings: \
+            \n1) url - get the source code of the url specified using --url/-u.\
+            \n2) archive_url - get the source code of the newest archive for the supplied url, alias of newest.\
+            \n3) oldest - get the source code of the oldest archive for the supplied url.\
+            \n4) newest - get the source code of the newest archive for the supplied url.\
+            \n5) save - Create a new archive and get the source code of this new archive for the supplied url."
+
+    return output

-    return ("Use get as \"--get 'source'\", 'source' can be one of the followings: \
-        \n1) url - get the source code of the url specified using --url/-u.\
-        \n2) oldest - get the source code of the oldest archive for the supplied url.\
-        \n3) newest - get the source code of the newest archive for the supplied url.\
-        \n4) save - Create a new archive and get the source code of this new archive for the supplied url.")

 def args_handler(args):
    if args.version:
-        return ("waybackpy version %s" % __version__)
+        return "waybackpy version %s" % __version__

    if not args.url:
-        return ("waybackpy %s \nSee 'waybackpy --help' for help using this tool." % __version__)
+        return (
+            "waybackpy %s \nSee 'waybackpy --help' for help using this tool."
+            % __version__
+        )

+    obj = Url(args.url)
    if args.user_agent:
        obj = Url(args.url, args.user_agent)
-    else:
-        obj = Url(args.url)

    if args.save:
-        return _save(obj)
-    if args.oldest:
-        return _oldest(obj)
-    if args.newest:
-        return _newest(obj)
-    if args.known_urls:
-        return _known_urls(obj, args)
-    if args.total:
-        return _total_archives(obj)
-    if args.near:
-        return _near(obj, args)
-    if args.get:
-        return _get(obj, args)
-    return ("You only specified the URL. But you also need to specify the operation.\nSee 'waybackpy --help' for help using this tool.")
+        output = _save(obj)
+    elif args.archive_url:
+        output = _archive_url(obj)
+    elif args.json:
+        output = _json(obj)
+    elif args.oldest:
+        output = _oldest(obj)
+    elif args.newest:
+        output = _newest(obj)
+    elif args.known_urls:
+        output = _known_urls(obj, args)
+    elif args.total:
+        output = _total_archives(obj)
+    elif args.near:
+        output = _near(obj, args)
+    elif args.get:
+        output = _get(obj, args)
+    else:
+        output = (
+            "You only specified the URL. But you also need to specify the operation."
+            "\nSee 'waybackpy --help' for help using this tool."
+        )
+    return output
+

 def parse_args(argv):
    parser = argparse.ArgumentParser()

-    requiredArgs = parser.add_argument_group('URL argument (required)')
-    requiredArgs.add_argument("--url", "-u", help="URL on which Wayback machine operations would occur")
+    requiredArgs = parser.add_argument_group("URL argument (required)")
+    requiredArgs.add_argument(
+        "--url", "-u", help="URL on which Wayback machine operations would occur"
+    )
+
+    userAgentArg = parser.add_argument_group("User Agent")
+    help_text = 'User agent, default user_agent is "waybackpy python package - https://github.com/akamhy/waybackpy"'
+    userAgentArg.add_argument("--user_agent", "-ua", help=help_text)

-    userAgentArg = parser.add_argument_group('User Agent')
-    userAgentArg.add_argument("--user_agent", "-ua", help="User agent, default user_agent is \"waybackpy python package - https://github.com/akamhy/waybackpy\"")
-    
    saveArg = parser.add_argument_group("Create new archive/save URL")
-    saveArg.add_argument("--save", "-s", action='store_true', help="Save the URL on the Wayback machine")
-    
+    saveArg.add_argument(
+        "--save", "-s", action="store_true", help="Save the URL on the Wayback machine"
+    )
+
+    auArg = parser.add_argument_group("Get the latest Archive")
+    auArg.add_argument(
+        "--archive_url",
+        "-au",
+        action="store_true",
+        help="Get the latest archive URL, alias for --newest",
+    )
+
+    jsonArg = parser.add_argument_group("Get the JSON data")
+    jsonArg.add_argument(
+        "--json",
+        "-j",
+        action="store_true",
+        help="JSON data of the availability API request",
+    )
+
    oldestArg = parser.add_argument_group("Oldest archive")
-    oldestArg.add_argument("--oldest", "-o", action='store_true', help="Oldest archive for the specified URL")
-    
+    oldestArg.add_argument(
+        "--oldest",
+        "-o",
+        action="store_true",
+        help="Oldest archive for the specified URL",
+    )
+
    newestArg = parser.add_argument_group("Newest archive")
-    newestArg.add_argument("--newest", "-n", action='store_true', help="Newest archive for the specified URL")
-    
+    newestArg.add_argument(
+        "--newest",
+        "-n",
+        action="store_true",
+        help="Newest archive for the specified URL",
+    )
+
    totalArg = parser.add_argument_group("Total number of archives")
-    totalArg.add_argument("--total", "-t", action='store_true', help="Total number of archives for the specified URL")
-    
+    totalArg.add_argument(
+        "--total",
+        "-t",
+        action="store_true",
+        help="Total number of archives for the specified URL",
+    )
+
    getArg = parser.add_argument_group("Get source code")
-    getArg.add_argument("--get", "-g", help="Prints the source code of the supplied url. Use '--get help' for extended usage")
+    getArg.add_argument(
+        "--get",
+        "-g",
+        help="Prints the source code of the supplied url. Use '--get help' for extended usage",
+    )

-    knownUrlArg = parser.add_argument_group("URLs known and archived to Waybcak Machine for the site.")
-    knownUrlArg.add_argument("--known_urls", "-ku", action='store_true', help="URLs known for the domain.")
-    knownUrlArg.add_argument("--subdomain", "-sub", action='store_true', help="Use with '--known_urls' to include known URLs for subdomains.")
-    knownUrlArg.add_argument("--alive", "-a", action='store_true', help="Only include live URLs. Will not inlclude dead links.")
+    knownUrlArg = parser.add_argument_group(
+        "URLs known and archived to Waybcak Machine for the site."
+    )
+    knownUrlArg.add_argument(
+        "--known_urls", "-ku", action="store_true", help="URLs known for the domain."
+    )
+    help_text = "Use with '--known_urls' to include known URLs for subdomains."
+    knownUrlArg.add_argument("--subdomain", "-sub", action="store_true", help=help_text)
+    help_text = "Only include live URLs. Will not inlclude dead links."
+    knownUrlArg.add_argument("--alive", "-a", action="store_true", help=help_text)

+    nearArg = parser.add_argument_group("Archive close to time specified")
+    nearArg.add_argument(
+        "--near", "-N", action="store_true", help="Archive near specified time"
+    )

-    nearArg = parser.add_argument_group('Archive close to time specified')
-    nearArg.add_argument("--near", "-N", action='store_true', help="Archive near specified time")
-
-    nearArgs = parser.add_argument_group('Arguments that are used only with --near')
+    nearArgs = parser.add_argument_group("Arguments that are used only with --near")
    nearArgs.add_argument("--year", "-Y", type=int, help="Year in integer")
    nearArgs.add_argument("--month", "-M", type=int, help="Month in integer")
    nearArgs.add_argument("--day", "-D", type=int, help="Day in integer.")
    nearArgs.add_argument("--hour", "-H", type=int, help="Hour in intege")
    nearArgs.add_argument("--minute", "-MIN", type=int, help="Minute in integer")

-    parser.add_argument("--version", "-v", action='store_true', help="Waybackpy version")
-    
+    parser.add_argument(
+        "--version", "-v", action="store_true", help="Waybackpy version"
+    )
+
    return parser.parse_args(argv[1:])

+
 def main(argv=None):
    if argv is None:
        argv = sys.argv
@ -161,5 +300,6 @@ def main(argv=None):
    output = args_handler(args)
    print(output)

+
 if __name__ == "__main__":
    sys.exit(main(sys.argv))
--- a/waybackpy/exceptions.py
+++ b/waybackpy/exceptions.py
@ -1,6 +1,13 @@
 # -*- coding: utf-8 -*-

+
 class WaybackError(Exception):
    """
-    Raised when API Service error.
+    Raised when Wayback Machine API Service is unreachable/down.
+    """
+
+
+class URLError(Exception):
+    """
+    Raised when malformed URLs are passed as arguments.
    """
--- a/waybackpy/wrapper.py
+++ b/waybackpy/wrapper.py
@ -1,17 +1,12 @@
 # -*- coding: utf-8 -*-

 import re
-import sys
-import json
-from datetime import datetime
-from waybackpy.exceptions import WaybackError
+from datetime import datetime, timedelta
+from waybackpy.exceptions import WaybackError, URLError
 from waybackpy.__version__ import __version__
+import requests
+import concurrent.futures

-if sys.version_info >= (3, 0):  # If the python ver >= 3
-    from urllib.request import Request, urlopen
-    from urllib.error import URLError
-else:  # For python2.x
-    from urllib2 import Request, urlopen, URLError

 default_UA = "waybackpy python package - https://github.com/akamhy/waybackpy"

@ -19,9 +14,7 @@ default_UA = "waybackpy python package - https://github.com/akamhy/waybackpy"
 def _archive_url_parser(header):
    """Parse out the archive from header."""
    # Regex1
-    arch = re.search(
-        r"Content-Location: (/web/[0-9]{14}/.*)", str(header)
-    )
+    arch = re.search(r"Content-Location: (/web/[0-9]{14}/.*)", str(header))
    if arch:
        return "web.archive.org" + arch.group(1)
    # Regex2
@ -36,7 +29,7 @@ def _archive_url_parser(header):
        return arch.group(1)
    raise WaybackError(
        "No archive URL found in the API response. "
-        "This version of waybackpy (%s) is likely out of date. Visit "
+        "This version of waybackpy (%s) is likely out of date or WayBack Machine is malfunctioning. Visit "
        "https://github.com/akamhy/waybackpy for the latest version "
        "of waybackpy.\nHeader:\n%s" % (__version__, str(header))
    )
@ -49,19 +42,21 @@ def _wayback_timestamp(**kwargs):
    )


-def _get_response(req):
+def _get_response(endpoint, params=None, headers=None):
    """Get response for the supplied request."""
+
    try:
-        response = urlopen(req)  # nosec
+        response = requests.get(endpoint, params=params, headers=headers)
    except Exception:
        try:
-            response = urlopen(req)  # nosec
+            response = requests.get(endpoint, params=params, headers=headers)  # nosec
        except Exception as e:
-            exc = WaybackError("Error while retrieving %s" % req.full_url)
+            exc = WaybackError("Error while retrieving %s" % endpoint)
            exc.__cause__ = e
            raise exc
    return response

+
 class Url:
    """waybackpy Url object"""

@ -69,58 +64,138 @@ class Url:
        self.url = url
        self.user_agent = user_agent
        self._url_check()  # checks url validity on init.
+        self._archive_url = None  # URL of archive
+        self.timestamp = None  # timestamp for last archive
+        self._alive_url_list = []

    def __repr__(self):
        return "waybackpy.Url(url=%s, user_agent=%s)" % (self.url, self.user_agent)

    def __str__(self):
-        return "%s" % self._clean_url()
+        if not self._archive_url:
+            self._archive_url = self.archive_url
+        return "%s" % self._archive_url

    def __len__(self):
-        return len(self._clean_url())
+        td_max = timedelta(
+            days=999999999, hours=23, minutes=59, seconds=59, microseconds=999999
+        )
+
+        if not self.timestamp:
+            self.timestamp = self._timestamp
+
+        if self.timestamp == datetime.max:
+            return td_max.days
+
+        diff = datetime.utcnow() - self.timestamp
+        return diff.days

    def _url_check(self):
        """Check for common URL problems."""
        if "." not in self.url:
            raise URLError("'%s' is not a vaild URL." % self.url)

+    @property
+    def JSON(self):
+        """
+        Returns JSON data from 'https://archive.org/wayback/available?url=YOUR-URL'.
+        """
+        endpoint = "https://archive.org/wayback/available"
+        headers = {"User-Agent": "%s" % self.user_agent}
+        payload = {"url": "%s" % self._clean_url()}
+        response = _get_response(endpoint, params=payload, headers=headers)
+        return response.json()
+
+    @property
+    def archive_url(self):
+        """
+        Returns any random archive for the instance.
+        But if near, oldest, newest were used before
+        then it returns the same archive again.
+
+        We cache archive in self._archive_url
+        """
+
+        if self._archive_url:
+            return self._archive_url
+
+        data = self.JSON
+
+        if not data["archived_snapshots"]:
+            archive_url = None
+        else:
+            archive_url = data["archived_snapshots"]["closest"]["url"]
+            archive_url = archive_url.replace(
+                "http://web.archive.org/web/", "https://web.archive.org/web/", 1
+            )
+        self._archive_url = archive_url
+        return archive_url
+
+    @property
+    def _timestamp(self):
+        """
+        Get timestamp of last fetched archive.
+        If used before fetching any archive, This
+        randomly picks archive.
+        """
+
+        if self.timestamp:
+            return self.timestamp
+
+        data = self.JSON
+
+        if not data["archived_snapshots"]:
+            ts = datetime.max
+
+        else:
+            ts = datetime.strptime(
+                data["archived_snapshots"]["closest"]["timestamp"], "%Y%m%d%H%M%S"
+            )
+        self.timestamp = ts
+        return ts
+
    def _clean_url(self):
-        """Fix the URL, if possible."""
+        """
+        Remove newlines
+        replace " " with "_"
+        """
        return str(self.url).strip().replace(" ", "_")

    def save(self):
        """Create a new Wayback Machine archive for this URL."""
        request_url = "https://web.archive.org/save/" + self._clean_url()
-        hdr = {"User-Agent": "%s" % self.user_agent}  # nosec
-        req = Request(request_url, headers=hdr)  # nosec
-        header = _get_response(req).headers
-        return "https://" + _archive_url_parser(header)
+        headers = {"User-Agent": "%s" % self.user_agent}
+        response = _get_response(request_url, params=None, headers=headers)
+        self._archive_url = "https://" + _archive_url_parser(response.headers)
+        self.timestamp = datetime.utcnow()
+        return self

    def get(self, url="", user_agent="", encoding=""):
        """Return the source code of the supplied URL.
        If encoding is not supplied, it is auto-detected from the response.
        """
-        
+
        if not url:
            url = self._clean_url()

        if not user_agent:
            user_agent = self.user_agent

-        hdr = {"User-Agent": "%s" % user_agent}
-        req = Request(url, headers=hdr)  # nosec
-        response = _get_response(req)
+        headers = {"User-Agent": "%s" % self.user_agent}
+        response = _get_response(url, params=None, headers=headers)
+
        if not encoding:
            try:
-                encoding = response.headers["content-type"].split("charset=")[-1]
+                encoding = response.encoding
            except AttributeError:
                encoding = "UTF-8"
-        return response.read().decode(encoding.replace("text/html", "UTF-8", 1))
+
+        return response.content.decode(encoding.replace("text/html", "UTF-8", 1))

    def near(self, year=None, month=None, day=None, hour=None, minute=None):
-        """ Return the closest Wayback Machine archive to the time supplied.
-            Supported params are year, month, day, hour and minute.
-            Any non-supplied parameters default to the current time.
+        """Return the closest Wayback Machine archive to the time supplied.
+        Supported params are year, month, day, hour and minute.
+        Any non-supplied parameters default to the current time.

        """
        now = datetime.utcnow().timetuple()
@ -132,25 +207,27 @@ class Url:
            minute=minute if minute else now.tm_min,
        )

-        request_url = "https://archive.org/wayback/available?url=%s&timestamp=%s" % (
-            self._clean_url(),
-            timestamp,
-        )
-        hdr = {"User-Agent": "%s" % self.user_agent}
-        req = Request(request_url, headers=hdr)  # nosec
-        response = _get_response(req)
-        data = json.loads(response.read().decode("UTF-8"))
+        endpoint = "https://archive.org/wayback/available"
+        headers = {"User-Agent": "%s" % self.user_agent}
+        payload = {"url": "%s" % self._clean_url(), "timestamp": timestamp}
+        response = _get_response(endpoint, params=payload, headers=headers)
+        data = response.json()
        if not data["archived_snapshots"]:
            raise WaybackError(
                "Can not find archive for '%s' try later or use wayback.Url(url, user_agent).save() "
                "to create a new archive." % self._clean_url()
            )
        archive_url = data["archived_snapshots"]["closest"]["url"]
-        # wayback machine returns http sometimes, idk why? But they support https
        archive_url = archive_url.replace(
            "http://web.archive.org/web/", "https://web.archive.org/web/", 1
        )
-        return archive_url
+
+        self._archive_url = archive_url
+        self.timestamp = datetime.strptime(
+            data["archived_snapshots"]["closest"]["timestamp"], "%Y%m%d%H%M%S"
+        )
+
+        return self

    def oldest(self, year=1994):
        """Return the oldest Wayback Machine archive for this URL."""
@ -166,22 +243,36 @@ class Url:

    def total_archives(self):
        """Returns the total number of Wayback Machine archives for this URL."""
-        hdr = {"User-Agent": "%s" % self.user_agent}
-        request_url = (
-            "https://web.archive.org/cdx/search/cdx?url=%s&output=json&fl=statuscode"
-            % self._clean_url()
-        )
-        req = Request(request_url, headers=hdr)  # nosec
-        response = _get_response(req)
+
+        endpoint = "https://web.archive.org/cdx/search/cdx"
+        headers = {
+            "User-Agent": "%s" % self.user_agent,
+            "output": "json",
+            "fl": "statuscode",
+        }
+        payload = {"url": "%s" % self._clean_url()}
+        response = _get_response(endpoint, params=payload, headers=headers)
+
        # Most efficient method to count number of archives (yet)
-        return str(response.read()).count(",")
+        return response.text.count(",")
+
+    def live_urls_picker(self, url):
+
+        try:
+            response_code = requests.get(url).status_code
+        except Exception:
+            return  # we don't care if urls are not opening
+
+        # 200s are OK and 300s are usually redirects, if you don't want redirects replace 400 with 300
+        if response_code >= 400:
+            return
+
+        self._alive_url_list.append(url)

    def known_urls(self, alive=False, subdomain=False):
        """Returns list of URLs known to exist for given domain name
        because these URLs were crawled by WayBack Machine bots.
-
        Useful for pen-testers and others.
-
        Idea by Mohammed Diaa (https://github.com/mhmdiaa) from:
        https://gist.github.com/mhmdiaa/adf6bff70142e5091792841d4b372050
        """
@ -190,35 +281,24 @@ class Url:

        if subdomain:
            request_url = (
-            "https://web.archive.org/cdx/search/cdx?url=*.%s/*&output=json&fl=original&collapse=urlkey" 
-            % self._clean_url()
+                "https://web.archive.org/cdx/search/cdx?url=*.%s/*&output=json&fl=original&collapse=urlkey"
+                % self._clean_url()
            )
-
        else:
            request_url = (
-            "http://web.archive.org/cdx/search/cdx?url=%s/*&output=json&fl=original&collapse=urlkey" 
-            % self._clean_url()
+                "http://web.archive.org/cdx/search/cdx?url=%s/*&output=json&fl=original&collapse=urlkey"
+                % self._clean_url()
            )

-        hdr = {"User-Agent": "%s" % self.user_agent}
-        req = Request(request_url, headers=hdr)  # nosec
-        response = _get_response(req)
-
-        data = json.loads(response.read().decode("UTF-8"))
+        headers = {"User-Agent": "%s" % self.user_agent}
+        response = _get_response(request_url, params=None, headers=headers)
+        data = response.json()
        url_list = [y[0] for y in data if y[0] != "original"]

-        #Remove all deadURLs from url_list if alive=True
+        # Remove all deadURLs from url_list if alive=True
        if alive:
-            tmp_url_list = []
-            for url in url_list:
+            with concurrent.futures.ThreadPoolExecutor() as executor:
+                executor.map(self.live_urls_picker, url_list)
+            url_list = self._alive_url_list

-                try:
-                    urlopen(url)
-                except:
-                    continue
-
-                tmp_url_list.append(url)
-
-            url_list = tmp_url_list
-        
-        return url_list
+        return url_list
Author	SHA1	Message	Date
Akash Mahanty	7c7fd75376	No need to fetch archive_url and timestamp from availability API on init (#55 ) * No need to fetch archive_url and timestamp from availability API on init. Not useful if all I want is to archive a page * Update test_wrapper.py * Update wrapper.py * Update test_wrapper.py * Update wrapper.py * Update cli.py * Update wrapper.py * Update __version__.py * Update __version__.py * Update __version__.py * Update __version__.py * Update setup.py * Update README.md	2021-01-02 11:10:23 +05:30
Akash Mahanty	0b71433667	v2.3.1 (#54 ) * 2.3.1 * 2.3.1	2021-01-01 19:15:23 +05:30
Akash Mahanty	1b499a7594	removed JSON from init, this was resulting in too much unnecessary taffic. Some users who are thousands of URLs were blocked by IA (#53 ) closes #52	2021-01-01 16:38:57 +05:30
Akash Mahanty	da390ee8a3	improve maintainability and reduce code cognitive complexity (#49 )	2020-12-15 10:24:13 +05:30
Akash Mahanty	d3e68d0e70	code formated with black (#47 )	2020-12-14 01:18:04 +05:30
Akash Mahanty	fde28d57aa	Update CONTRIBUTING.md	2020-12-14 00:16:29 +05:30
Akash Mahanty	6092e504c8	Update CONTRIBUTING.md	2020-12-14 00:15:51 +05:30
Akash Mahanty	93ef60ecd2	v2.3.0 (#46 ) * v2.3.0 * v2.3.0 * decrease line length	2020-12-14 00:14:54 +05:30
Akash Mahanty	461b3f74c9	UPDATE header image url	2020-12-13 23:09:59 +05:30
Akash Mahanty	3c53b411b0	Improve the appearance of readme (#45 ) * replaced text header wth image * svg * Update README.md * Update README.md * Update README.md * level 2 * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Create CONTRIBUTING.md * Update README.md * Add files via upload * Update README.md * Delete waybackpy-colored 284.png * Delete waybackpy colored.png * Update README.md * Update index.rst * Update index.rst * Update index.rst * Update setup.py * Delete index.rst * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md	2020-12-13 23:08:16 +05:30
pyup.io bot	8125526061	create pyup.io config file (#44 )	2020-12-13 22:31:49 +05:30
Akash Mahanty	2dc81569a8	Create .pep8speaks.yml	2020-12-13 17:58:09 +05:30
Akash Mahanty	fd163f3d36	Update wrapper.py	2020-12-13 17:12:32 +05:30
Akash Mahanty	a0a918cf0d	.	2020-12-13 17:10:28 +05:30
Akash Mahanty	4943cf6873	remove print stmnt, update ci	2020-12-13 16:37:35 +05:30
Akash Mahanty	bc3efc7d63	now using requests lib as it handles errors nicely (#42 ) * now using requests lib as it handles errors nicely * remove unused import (urllib) * FIX : replaced full_url with endpoint (not using urlib) * LINT : Found in waybackpy\wrapper.py:88 Unnecessary else after return	2020-12-13 15:44:37 +05:30
Akash Mahanty	f89368f16d	LINT : Found in waybackpy\wrapper.py:88 Unnecessary else after return	2020-12-13 15:39:23 +05:30
Akash Mahanty	c919a6a605	FIX : replaced full_url with endpoint (not using urlib)	2020-12-13 15:22:56 +05:30
Akash Mahanty	0280fca189	remove unused import (urllib)	2020-12-13 15:13:51 +05:30
Akash Mahanty	60ee8b95a8	now using requests lib as it handles errors nicely	2020-12-13 15:05:57 +05:30
Akash Mahanty	ca51c14332	deleted .travis.yml, link with flake (#41 ) close #38	2020-11-26 13:06:50 +05:30
Akash Mahanty	525cf17c6f	Update ci.yml	2020-11-26 12:14:15 +05:30
Akash Mahanty	406e03c52f	Update ci.yml	2020-11-26 12:04:45 +05:30
Akash Mahanty	672b33e83a	Update ci.yml	2020-11-26 10:10:10 +05:30
Akash Mahanty	b19b840628	Update ci.yml	2020-11-26 10:01:55 +05:30
Akash Mahanty	a6df4f899c	Update ci.yml	2020-11-26 09:26:11 +05:30
Akash Mahanty	7686e9c20d	Update README.md (#40 )	2020-11-26 09:18:26 +05:30
Akash Mahanty	3c5932bc39	now using gh actions (#39 )	2020-11-26 09:09:53 +05:30
Akash Mahanty	f9a986f489	Create ci.yml	2020-11-26 08:55:23 +05:30
Akash Mahanty	0d7458ee90	per https://docs.travis-ci.com/user/languages/python/, Python builds are not available on the macOS	2020-11-26 08:08:59 +05:30
Akash Mahanty	ac8b9d6a50	use osx, huge backlog on .org travis for linux builds	2020-11-26 08:03:27 +05:30
Akash Mahanty	58cd9c28e7	Threading enabled checking for URLs	2020-11-26 06:15:42 +05:30
Akash Mahanty	5088305a58	removed python2 compatibility code	2020-11-21 17:00:11 +05:30
Akash Mahanty	9f847a5e55	change pepy.tech download count link, they removed the month page	2020-11-11 10:44:14 +05:30
Akash Mahanty	6c04c2f3d3	+ https://github.com/akamhy/waybackpy/graphs/contributors	2020-11-04 08:09:30 +05:30
Akash Mahanty	925be7b17e	V2.2.0	2020-10-17 17:10:46 +05:30
Akash Mahanty	2b132456ac	updated index.rst and minor docs updated.	2020-10-17 16:56:51 +05:30
Akash Mahanty	50e3154a4e	lint README.md	2020-10-17 12:01:49 +05:30
Akash Mahanty	7aef50428f	add link to the repo	2020-10-17 11:51:56 +05:30
Akash Mahanty	d8ec0f5025	More pythonic code snippets in README (#36 )	2020-10-17 11:49:27 +05:30
Akash Mahanty	0a2f97c034	Update README, drop python 2 support * Drop python 2 support * updated docs * added new docs	2020-10-16 22:37:32 +05:30
Akash Mahanty	3e9cf23578	3.9 archive doesn't not exist yet.	2020-10-16 19:43:06 +05:30
Akash Mahanty	7f927ec7be	added tests for json and archive_url, updated broken tests (#34 ) * added tests for json and archive_url, updated broken tests * drop 2.7 support	2020-10-16 19:25:45 +05:30
Akash Mahanty	9de6393cd5	Add support for JSON and archive_url (#33 ) CLI support for JSON and archive_url attributes	2020-10-16 15:16:18 +05:30
danvalen1	91e7f65617	Fixing len() bug (#32 ) * added class functionality * Update wrapper.py * style edits * fixed bug with len() of url() * fixing len() bug * fixing len() bug * squashing bug * removed test notebook	2020-10-16 10:04:13 +05:30
danvalen1	d465454019	Adding attributes to Url class (#28 ) * added class functionality * Update wrapper.py * style edits	2020-10-15 22:10:32 +05:30
Akash Mahanty	1a81eb97fb	lint	2020-10-03 16:58:11 +05:30
Akash Mahanty	6b3b2e2a7d	tests for newly added known_urls feature	2020-10-03 09:33:50 +05:30
Akash Mahanty	82c65454e6	2.1.9	2020-10-03 01:34:15 +05:30
Akash Mahanty	19710461b6	Update setup.py	2020-10-03 01:33:46 +05:30
Akash Mahanty	a3661d6b85	Update index.rst	2020-10-03 01:33:15 +05:30
Akash Mahanty	58375e4ef4	fix broken links	2020-10-03 01:31:28 +05:30
Akash Mahanty	ea023e98da	update	2020-10-03 01:22:51 +05:30