Commit Graph

491 Commits

Author SHA1 Message Date
Akash Mahanty
5407681c34
v3.0.6 (#170)
* remove the license section from readme

This does not mean that I'm waving the copyrights rather just formatting the README

* remove useless external links form the README lead

and also added a line about the recentness of the newest method between the availability and CDX server API.

* incr version to 3.0.6 and change date to todays da

-te that is 15th of March, 2022.

* update secsi and DI section

* v3.0.5 --> v3.0.6
2022-03-15 20:33:51 +05:30
Akash Mahanty
cfd977135d
Update CITATION.cff (#169) 2022-03-04 11:48:49 +05:30
eggplants
7a5e0bfdaf
fix: cff format (#168) 2022-03-04 03:10:27 +05:30
eggplants
48dcda8020
add: typed marker (PEP561) (#167) 2022-03-03 19:05:43 +05:30
eggplants
3ed2170a32
add: CITATION.cff (#166) 2022-03-03 19:05:23 +05:30
Akash Mahanty
d6ef55020c
undo drop python3.6, see #162 (#163) 2022-02-18 21:54:33 +05:30
Akash Mahanty
2650943f9d
v3.0.4 (#160)
* Update README.md

* Update README.md

* update asciinema link

* v3.0.4

* update video link
2022-02-18 16:05:58 +05:30
Akash Mahanty
4b218d35cb
Cdx based oldest newest and near (#159)
* implement oldest newest and near methods in the cdx interface class, now cli uses the cdx methods instead of availablity api methods.

* handle the closest parameter derivative methods more efficiently and also handle exceptions gracefully.

* update test code
2022-02-18 13:17:40 +05:30
Akash Mahanty
f990b93f8a
Add sort, use_pagination and closest (#158)
* add sort param support in CDX API class

see https://nla.github.io/outbackcdx/api.html#operation/query

sort takes string input which must be one of the follwoing:
- default
- closest
- reverse

This commit shall help in closing issue at https://github.com/akamhy/waybackpy/issues/155

* add BlockedSiteError for cases when archiving is blocked by site's robots.txt

* create check_for_blocked_site for handling the BlockedSiteError for sites that are blocking wayback machine by their robots.txt policy

* add attrs use_pagination and closest, which are can be used to use the pagination API and lookup archive close to a timestamp respectively. And now to get out of infinte blank pages loop just check for two succesive black and not total two blank pages while using the CDX server API.

* added cli support for sort, use-pagination and closest

* added tests

* fix codeql warnings, nothing to worry about here.

* fix save test for archive_url
2022-02-18 00:24:14 +05:30
Akash Mahanty
3a44a710d3
add sort param support in CDX API class (#156)
see https://nla.github.io/outbackcdx/api.html#operation/query

sort takes string input which must be one of the follwoing:
- default
- closest
- reverse

This commit shall help in closing issue at https://github.com/akamhy/waybackpy/issues/155
2022-02-17 12:17:23 +05:30
Akash Mahanty
f63c6adf79 Trigger Build 2022-02-09 17:29:19 +05:30
eggplants
b4d3393ef1
fix: move metadata from __init__.py into setup.cfg (#153) 2022-02-09 17:20:23 +05:30
Akash Mahanty
cd5c3c61a5 fix imports with isort 2022-02-09 16:18:25 +05:30
Akash Mahanty
87fb5ecd58 remove latest version funcs from utils, they were unused. 2022-02-09 16:12:30 +05:30
Akash Mahanty
5954fcc646 format with black 2022-02-09 15:51:11 +05:30
Akash Mahanty
89016d433c added trove Typing :: Typed and Development Status :: 5 - Production/Stable 2022-02-09 15:47:38 +05:30
Akash Mahanty
edaa1d5d54 update value to the new limit. 2022-02-09 15:40:38 +05:30
Akash Mahanty
16f94db144 incr version to v3.0.3 2022-02-09 14:33:16 +05:30
Akash Mahanty
25eb709ade improve doc strings and comments and remove useless exceptions. 2022-02-09 14:32:15 +05:30
Akash Mahanty
6d233f24fc apply isort 2022-02-09 11:20:59 +05:30
Akash Mahanty
ec341fa8b3 refactor code in cli module 2022-02-09 11:20:10 +05:30
Akash Mahanty
cf18090f90 fix typo 2022-02-09 09:52:20 +05:30
Akash Mahanty
81162eebd0
issues with HN 2022-02-08 21:28:25 +05:30
Akash Mahanty
ca4f79a2e3
+ jfinkhaeuser and rafael (#150) 2022-02-08 20:34:34 +05:30
Akash Mahanty
27f2727049 add cli alias for --start-timestamp(--from) and --end-timestamp(--to) to conform with the CDX API docs. 2022-02-08 20:12:19 +05:30
Akash Mahanty
118dc6c523 add test for wrapper module 2022-02-08 20:08:44 +05:30
Akash Mahanty
1216ffbc70 lint and refactor cli module 2022-02-08 20:06:17 +05:30
Akash Mahanty
d58a5f0ee5 explicitly exculde some dirs from flake8 check 2022-02-08 18:59:13 +05:30
Akash Mahanty
7e7412d9d1 remove deepsource, LGTM is better and has fewer False Postives. 2022-02-08 18:49:44 +05:30
Akash Mahanty
70c38c5a60
+ codecov badge 2022-02-08 17:49:05 +05:30
Akash Mahanty
f8bf9c16f9
Add tests (#149)
* enable codecov

* fix save_urls_on_file

* increase the limit of CDX to 25000 from 5000. 5X increase.

* added test for the CLI module

* make flake 8 happy

* make mypy happy
2022-02-08 17:46:59 +05:30
Akash Mahanty
2bbfee7b2f
replace non-ASCII emojis with GitHub hosted equivalent images (#148) 2022-02-08 11:43:32 +05:30
deepsource-autofix[bot]
7317bd7183
Remove blank lines after docstring (#146)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2022-02-08 10:14:20 +05:30
deepsource-autofix[bot]
e0dfbe0b7d
Fix comparison constant position (#145)
* Fix comparison constant position

* format with black

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Akash Mahanty <akamhy@yahoo.com>
2022-02-08 10:06:23 +05:30
eggplants
0b631592ea
Improve pylint score (#142)
* fix: errors to improve pylint scores

* fix: test

* fix

* add: flake ignore rule to pip8speaks conf

* fix

* add: test patterns to deepsource conf
2022-02-08 06:42:20 +09:00
Akash Mahanty
d3a8f343f8
+ [eggplants](https://github.com/eggplants) (#143) 2022-02-08 01:41:10 +05:30
Akash Mahanty
97f8b96411
added docstrings, added some static type hints and also lint. (#141)
* added docstrings, added some static type hints and also lint.

* added doc strings and changed some internal variable names for more clarity.

* make flake8 happy

* add descriptive docstrings and type hints in waybackpy/cdx_snapshot.py

* remove useless code and add docstrings and also lint using pylint.

* remove unwarented test

* added docstrings, lint using pylint and add a raise on 509 SC

* added docstrings and lint with pylint

* lint

* add doc strings and lint

* add docstrings and lint
2022-02-07 19:40:37 +05:30
DeepSource Bot
004ff26196 Add .deepsource.toml 2022-02-07 12:55:57 +00:00
Akash Mahanty
a772c22431 explicitly tell pep8speaks that mll is 88. 2022-02-06 21:00:15 +05:30
Akash Mahanty
b79f1c471e
Merge pull request #135 from eggplants/fix_cli
Fix cli.py
2022-02-05 16:54:36 +05:30
Akash Mahanty
f49d67a411
Merge pull request #136 from eggplants/429_error
Add TooManyRequestsError
2022-02-05 11:28:27 +05:30
Akash Mahanty
ad8bd25633
added badge of codacy (#139) 2022-02-05 10:05:17 +05:30
eggplants
d2a3946425
fix: escape banner 2022-02-05 10:12:27 +09:00
eggplants
7b6401d59b
fix: delete useless conds 2022-02-05 06:20:03 +09:00
eggplants
ed6160c54f
add: TooManyRequestsError 2022-02-05 06:19:02 +09:00
eggplants
fcab19a40a
fix: cli
print error message to stderr and specify defaults of url
2022-02-05 05:55:04 +09:00
eggplants
5f3cd28046
Fix Pylint errors were pointed out by codacy (#133)
* fix: pylint errors were pointed out by codacy

* fix: line length

* fix: help text

* fix: revert

https://stackoverflow.com/a/64477857 makes cli unusable

* fix: cli error and refactor codes
2022-02-05 05:25:40 +09:00
Akash Mahanty
9d9cc3328b add .pep8speaks.yml, override deafult 2022-02-05 00:53:38 +05:30
Akash Mahanty
b69e4dff37
rename params of main in cli.py to avoid using built-ins (#132)
* rename params of main in cli.py to avoid using built-ins

* Fix Line 32:80: E501 line too long (102 > 79 characters)
2022-02-05 00:30:35 +05:30
eggplants
d8cabdfdb5
Typing (#128)
* fix: CI yml name

* add: mypy configuraion

* add: type annotation to waybackpy modules

* add: type annotation to test modules

* fix: mypy command

* add: types-requests to dev deps

* fix: disable max-line-length

* fix: move pytest.ini into setup.cfg

* add: urllib3 to deps

* fix: Retry (ref: https://github.com/python/typeshed/issues/6893)

* fix: f-string

* fix: shorten long lines

* add: staticmethod decorator to no-self-use methods

* fix: str(headers)->headers_str

* fix: error message

* fix: revert "str(headers)->headers_str" and ignore assignment CaseInsensitiveDict with str

* fix: mypy error
2022-02-05 03:23:36 +09:00