* implement oldest newest and near methods in the cdx interface class, now cli uses the cdx methods instead of availablity api methods.
* handle the closest parameter derivative methods more efficiently and also handle exceptions gracefully.
* update test code
* add sort param support in CDX API class
see https://nla.github.io/outbackcdx/api.html#operation/query
sort takes string input which must be one of the follwoing:
- default
- closest
- reverse
This commit shall help in closing issue at https://github.com/akamhy/waybackpy/issues/155
* add BlockedSiteError for cases when archiving is blocked by site's robots.txt
* create check_for_blocked_site for handling the BlockedSiteError for sites that are blocking wayback machine by their robots.txt policy
* add attrs use_pagination and closest, which are can be used to use the pagination API and lookup archive close to a timestamp respectively. And now to get out of infinte blank pages loop just check for two succesive black and not total two blank pages while using the CDX server API.
* added cli support for sort, use-pagination and closest
* added tests
* fix codeql warnings, nothing to worry about here.
* fix save test for archive_url
* enable codecov
* fix save_urls_on_file
* increase the limit of CDX to 25000 from 5000. 5X increase.
* added test for the CLI module
* make flake 8 happy
* make mypy happy
* Fix comparison constant position
* format with black
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Akash Mahanty <akamhy@yahoo.com>
* added docstrings, added some static type hints and also lint.
* added doc strings and changed some internal variable names for more clarity.
* make flake8 happy
* add descriptive docstrings and type hints in waybackpy/cdx_snapshot.py
* remove useless code and add docstrings and also lint using pylint.
* remove unwarented test
* added docstrings, lint using pylint and add a raise on 509 SC
* added docstrings and lint with pylint
* lint
* add doc strings and lint
* add docstrings and lint
* fix: pylint errors were pointed out by codacy
* fix: line length
* fix: help text
* fix: revert
https://stackoverflow.com/a/64477857 makes cli unusable
* fix: cli error and refactor codes
