Benjamin_Loison/waybackpy
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity
486 Commits 1 Branch 37 Tags
b541d8dc0eb4b282e9c7b2f5efe4a04a67f05130
Commit Graph

486 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Akash Mahanty
b541d8dc0e added cli support for sort, use-pagination and closest 2022-02-17 21:15:05 +05:30
Akash Mahanty
0e7255c1d8 add attrs use_pagination and closest, which are can be used to use the pagination API and lookup archive close to a timestamp respectively. And now to get out of infinte blank pages loop just check for two succesive black and not total two blank pages while using the CDX server API. 2022-02-17 21:13:42 +05:30
Akash Mahanty
28f4b579f2 create check_for_blocked_site for handling the BlockedSiteError for sites that are blocking wayback machine by their robots.txt policy 2022-02-17 21:07:05 +05:30
Akash Mahanty
faa97e5877 add BlockedSiteError for cases when archiving is blocked by site's robots.txt 2022-02-17 21:04:21 +05:30
Akash Mahanty
b4adf4a372 add sort param support in CDX API class 
see https://nla.github.io/outbackcdx/api.html#operation/query

sort takes string input which must be one of the follwoing:
- default
- closest
- reverse

This commit shall help in closing issue at https://github.com/akamhy/waybackpy/issues/155
 2022-02-17 12:08:48 +05:30
Akash Mahanty
f63c6adf79 Trigger Build 2022-02-09 17:29:19 +05:30
eggplants
b4d3393ef1 fix: move metadata from __init__.py into setup.cfg (#153) 2022-02-09 17:20:23 +05:30
Akash Mahanty
cd5c3c61a5 fix imports with isort 3.0.3 2022-02-09 16:18:25 +05:30
Akash Mahanty
87fb5ecd58 remove latest version funcs from utils, they were unused. 2022-02-09 16:12:30 +05:30
Akash Mahanty
5954fcc646 format with black 2022-02-09 15:51:11 +05:30
Akash Mahanty
89016d433c added trove Typing :: Typed and Development Status :: 5 - Production/Stable 2022-02-09 15:47:38 +05:30
Akash Mahanty
edaa1d5d54 update value to the new limit. 2022-02-09 15:40:38 +05:30
Akash Mahanty
16f94db144 incr version to v3.0.3 2022-02-09 14:33:16 +05:30
Akash Mahanty
25eb709ade improve doc strings and comments and remove useless exceptions. 2022-02-09 14:32:15 +05:30
Akash Mahanty
6d233f24fc apply isort 2022-02-09 11:20:59 +05:30
Akash Mahanty
ec341fa8b3 refactor code in cli module 2022-02-09 11:20:10 +05:30
Akash Mahanty
cf18090f90 fix typo 2022-02-09 09:52:20 +05:30
Akash Mahanty
81162eebd0 issues with HN 2022-02-08 21:28:25 +05:30
Akash Mahanty
ca4f79a2e3 + jfinkhaeuser and rafael (#150) 2022-02-08 20:34:34 +05:30
Akash Mahanty
27f2727049 add cli alias for --start-timestamp(--from) and --end-timestamp(--to) to conform with the CDX API docs. 2022-02-08 20:12:19 +05:30
Akash Mahanty
118dc6c523 add test for wrapper module 2022-02-08 20:08:44 +05:30
Akash Mahanty
1216ffbc70 lint and refactor cli module 2022-02-08 20:06:17 +05:30
Akash Mahanty
d58a5f0ee5 explicitly exculde some dirs from flake8 check 2022-02-08 18:59:13 +05:30
Akash Mahanty
7e7412d9d1 remove deepsource, LGTM is better and has fewer False Postives. 2022-02-08 18:49:44 +05:30
Akash Mahanty
70c38c5a60 + codecov badge 2022-02-08 17:49:05 +05:30
Akash Mahanty
f8bf9c16f9 Add tests (#149) 
* enable codecov

* fix save_urls_on_file

* increase the limit of CDX to 25000 from 5000. 5X increase.

* added test for the CLI module

* make flake 8 happy

* make mypy happy
 2022-02-08 17:46:59 +05:30
Akash Mahanty
2bbfee7b2f replace non-ASCII emojis with GitHub hosted equivalent images (#148) 2022-02-08 11:43:32 +05:30
deepsource-autofix[bot]
7317bd7183 Remove blank lines after docstring (#146) 
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
 2022-02-08 10:14:20 +05:30
deepsource-autofix[bot]
e0dfbe0b7d Fix comparison constant position (#145) 
* Fix comparison constant position

* format with black

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Akash Mahanty <akamhy@yahoo.com>
 2022-02-08 10:06:23 +05:30
eggplants
0b631592ea Improve pylint score (#142) 
* fix: errors to improve pylint scores

* fix: test

* fix

* add: flake ignore rule to pip8speaks conf

* fix

* add: test patterns to deepsource conf
 2022-02-08 06:42:20 +09:00
Akash Mahanty
d3a8f343f8 + [eggplants](https://github.com/eggplants) (#143) 2022-02-08 01:41:10 +05:30
Akash Mahanty
97f8b96411 added docstrings, added some static type hints and also lint. (#141) 
* added docstrings, added some static type hints and also lint.

* added doc strings and changed some internal variable names for more clarity.

* make flake8 happy

* add descriptive docstrings and type hints in waybackpy/cdx_snapshot.py

* remove useless code and add docstrings and also lint using pylint.

* remove unwarented test

* added docstrings, lint using pylint and add a raise on 509 SC

* added docstrings and lint with pylint

* lint

* add doc strings and lint

* add docstrings and lint
 2022-02-07 19:40:37 +05:30
DeepSource Bot
004ff26196 Add .deepsource.toml 2022-02-07 12:55:57 +00:00
Akash Mahanty
a772c22431 explicitly tell pep8speaks that mll is 88. 2022-02-06 21:00:15 +05:30
Akash Mahanty
b79f1c471e Merge pull request #135 from eggplants/fix_cli 
Fix cli.py
 2022-02-05 16:54:36 +05:30
Akash Mahanty
f49d67a411 Merge pull request #136 from eggplants/429_error 
Add TooManyRequestsError
 2022-02-05 11:28:27 +05:30
Akash Mahanty
ad8bd25633 added badge of codacy (#139) 2022-02-05 10:05:17 +05:30
eggplants
d2a3946425 fix: escape banner 2022-02-05 10:12:27 +09:00
eggplants
7b6401d59b fix: delete useless conds 2022-02-05 06:20:03 +09:00
eggplants
ed6160c54f add: TooManyRequestsError 2022-02-05 06:19:02 +09:00
eggplants
fcab19a40a fix: cli 
print error message to stderr and specify defaults of url
 2022-02-05 05:55:04 +09:00
eggplants
5f3cd28046 Fix Pylint errors were pointed out by codacy (#133) 
* fix: pylint errors were pointed out by codacy

* fix: line length

* fix: help text

* fix: revert

https://stackoverflow.com/a/64477857 makes cli unusable

* fix: cli error and refactor codes
 2022-02-05 05:25:40 +09:00
Akash Mahanty
9d9cc3328b add .pep8speaks.yml, override deafult 2022-02-05 00:53:38 +05:30
Akash Mahanty
b69e4dff37 rename params of main in cli.py to avoid using built-ins (#132) 
* rename params of main in cli.py to avoid using built-ins

* Fix Line 32:80: E501 line too long (102 > 79 characters)
 2022-02-05 00:30:35 +05:30
eggplants
d8cabdfdb5 Typing (#128) 
* fix: CI yml name

* add: mypy configuraion

* add: type annotation to waybackpy modules

* add: type annotation to test modules

* fix: mypy command

* add: types-requests to dev deps

* fix: disable max-line-length

* fix: move pytest.ini into setup.cfg

* add: urllib3 to deps

* fix: Retry (ref: https://github.com/python/typeshed/issues/6893)

* fix: f-string

* fix: shorten long lines

* add: staticmethod decorator to no-self-use methods

* fix: str(headers)->headers_str

* fix: error message

* fix: revert "str(headers)->headers_str" and ignore assignment CaseInsensitiveDict with str

* fix: mypy error
 2022-02-05 03:23:36 +09:00
eggplants
320ef30371 fix: format md and yml (#129) 2022-02-04 22:31:46 +05:30
eggplants
e61447effd Format and lint codes and fix packaging (#125) 
* add: configure files (setup.py->setup.py+setup.cfg+pyproject.toml)

* add: __download_url__

* format with black and isort

* fix: flake8 section in setup.cfg

* add: E501 to flake ignore

* fix: metadata.name does not accept attr

* fix: merge __version__.py into __init__.py

* fix: flake8 errors in tests/

* fix: datetime.datetime -> datetime

* fix: banner

* fix: ignore W605 for banner

* fix: way to install deps in CI

* add: versem to setuptools

* fix: drop python<=3.6 (#126) from package and CI
 2022-02-03 19:13:39 +05:30
Akash Mahanty
947647f2e7 Merge pull request #124 from eggplants/fix_save_retry 
Fix save retry mechanism
 2022-02-03 18:01:51 +05:30
eggplants
bc1dc4dc96 fix: save retry mechanism 2022-02-03 19:45:16 +09:00
Akash Mahanty
5cbdfc040b waybackpy/cli.py : remove duplicate original_string from output_string in cdx 2022-01-30 21:02:25 +05:30