* implement oldest newest and near methods in the cdx interface class, now cli uses the cdx methods instead of availablity api methods.
* handle the closest parameter derivative methods more efficiently and also handle exceptions gracefully.
* update test code
* add sort param support in CDX API class
see https://nla.github.io/outbackcdx/api.html#operation/query
sort takes string input which must be one of the follwoing:
- default
- closest
- reverse
This commit shall help in closing issue at https://github.com/akamhy/waybackpy/issues/155
* add BlockedSiteError for cases when archiving is blocked by site's robots.txt
* create check_for_blocked_site for handling the BlockedSiteError for sites that are blocking wayback machine by their robots.txt policy
* add attrs use_pagination and closest, which are can be used to use the pagination API and lookup archive close to a timestamp respectively. And now to get out of infinte blank pages loop just check for two succesive black and not total two blank pages while using the CDX server API.
* added cli support for sort, use-pagination and closest
* added tests
* fix codeql warnings, nothing to worry about here.
* fix save test for archive_url
* enable codecov
* fix save_urls_on_file
* increase the limit of CDX to 25000 from 5000. 5X increase.
* added test for the CLI module
* make flake 8 happy
* make mypy happy
* Fix comparison constant position
* format with black
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Akash Mahanty <akamhy@yahoo.com>
* added docstrings, added some static type hints and also lint.
* added doc strings and changed some internal variable names for more clarity.
* make flake8 happy
* add descriptive docstrings and type hints in waybackpy/cdx_snapshot.py
* remove useless code and add docstrings and also lint using pylint.
* remove unwarented test
* added docstrings, lint using pylint and add a raise on 509 SC
* added docstrings and lint with pylint
* lint
* add doc strings and lint
* add docstrings and lint
* add: configure files (setup.py->setup.py+setup.cfg+pyproject.toml)
* add: __download_url__
* format with black and isort
* fix: flake8 section in setup.cfg
* add: E501 to flake ignore
* fix: metadata.name does not accept attr
* fix: merge __version__.py into __init__.py
* fix: flake8 errors in tests/
* fix: datetime.datetime -> datetime
* fix: banner
* fix: ignore W605 for banner
* fix: way to install deps in CI
* add: versem to setuptools
* fix: drop python<=3.6 (#126) from package and CI
* added RedirectSaveError - for failed saves if the URL is a permanent redirect.
* check if url is redirect before throwing exceptions, res.url is the redirect url if redirected at all
* update tests and cli errors
* Added some docstrings in utils.py
* renamed some func/meth to better names and added doc strings + lint
* added more docstrings
* more docstrings
* improve docstrings
* docstrings
* added more docstrings, lint
* fix import error
