Benjamin_Loison/waybackpy
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity
493 Commits 1 Branch 37 Tags
57f4be53d5d06476f76b4636c3da9492f200e398
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Akash Mahanty 57f4be53d5 ignore line 474 beacause 'error: <nothing> not callable'. 2022-03-29 10:24:55 +05:30
Akash Mahanty 64a4ce88af Minor copyediting and also deleted CONTRIBUTORS.md moved content to README.md 2022-03-29 03:39:50 +05:30
Akash MahantyandGitHub 5407681c34 v3.0.6 (#170) 
* remove the license section from readme

This does not mean that I'm waving the copyrights rather just formatting the README

* remove useless external links form the README lead

and also added a line about the recentness of the newest method between the availability and CDX server API.

* incr version to 3.0.6 and change date to todays da

-te that is 15th of March, 2022.

* update secsi and DI section

* v3.0.5 --> v3.0.6
 2022-03-15 20:33:51 +05:30
Akash MahantyandGitHub cfd977135d Update CITATION.cff (#169) 2022-03-04 11:48:49 +05:30
Akash MahantyandGitHub d6ef55020c undo drop python3.6, see #162 (#163) 2022-02-18 21:54:33 +05:30
Akash MahantyandGitHub 2650943f9d v3.0.4 (#160) 
* Update README.md

* Update README.md

* update asciinema link

* v3.0.4

* update video link
 2022-02-18 16:05:58 +05:30
Akash MahantyandGitHub 4b218d35cb Cdx based oldest newest and near (#159) 
* implement oldest newest and near methods in the cdx interface class, now cli uses the cdx methods instead of availablity api methods.

* handle the closest parameter derivative methods more efficiently and also handle exceptions gracefully.

* update test code
 2022-02-18 13:17:40 +05:30
Akash MahantyandGitHub f990b93f8a Add sort, use_pagination and closest (#158) 
* add sort param support in CDX API class

see https://nla.github.io/outbackcdx/api.html#operation/query

sort takes string input which must be one of the follwoing:
- default
- closest
- reverse

This commit shall help in closing issue at https://github.com/akamhy/waybackpy/issues/155

* add BlockedSiteError for cases when archiving is blocked by site's robots.txt

* create check_for_blocked_site for handling the BlockedSiteError for sites that are blocking wayback machine by their robots.txt policy

* add attrs use_pagination and closest, which are can be used to use the pagination API and lookup archive close to a timestamp respectively. And now to get out of infinte blank pages loop just check for two succesive black and not total two blank pages while using the CDX server API.

* added cli support for sort, use-pagination and closest

* added tests

* fix codeql warnings, nothing to worry about here.

* fix save test for archive_url
 2022-02-18 00:24:14 +05:30
Akash MahantyandGitHub 3a44a710d3 add sort param support in CDX API class (#156) 
see https://nla.github.io/outbackcdx/api.html#operation/query

sort takes string input which must be one of the follwoing:
- default
- closest
- reverse

This commit shall help in closing issue at https://github.com/akamhy/waybackpy/issues/155
 2022-02-17 12:17:23 +05:30
Akash Mahanty f63c6adf79 Trigger Build 2022-02-09 17:29:19 +05:30
Akash Mahanty cd5c3c61a5 fix imports with isort 2022-02-09 16:18:25 +05:30
Akash Mahanty 87fb5ecd58 remove latest version funcs from utils, they were unused. 2022-02-09 16:12:30 +05:30
Akash Mahanty 5954fcc646 format with black 2022-02-09 15:51:11 +05:30
Akash Mahanty 89016d433c added trove Typing :: Typed and Development Status :: 5 - Production/Stable 2022-02-09 15:47:38 +05:30
Akash Mahanty edaa1d5d54 update value to the new limit. 2022-02-09 15:40:38 +05:30
Akash Mahanty 16f94db144 incr version to v3.0.3 2022-02-09 14:33:16 +05:30
Akash Mahanty 25eb709ade improve doc strings and comments and remove useless exceptions. 2022-02-09 14:32:15 +05:30
Akash Mahanty 6d233f24fc apply isort 2022-02-09 11:20:59 +05:30
Akash Mahanty ec341fa8b3 refactor code in cli module 2022-02-09 11:20:10 +05:30
Akash Mahanty cf18090f90 fix typo 2022-02-09 09:52:20 +05:30
Akash MahantyandGitHub 81162eebd0 issues with HN 2022-02-08 21:28:25 +05:30
Akash MahantyandGitHub ca4f79a2e3 + jfinkhaeuser and rafael (#150) 2022-02-08 20:34:34 +05:30
Akash Mahanty 27f2727049 add cli alias for --start-timestamp(--from) and --end-timestamp(--to) to conform with the CDX API docs. 2022-02-08 20:12:19 +05:30
Akash Mahanty 118dc6c523 add test for wrapper module 2022-02-08 20:08:44 +05:30
Akash Mahanty 1216ffbc70 lint and refactor cli module 2022-02-08 20:06:17 +05:30
Akash Mahanty d58a5f0ee5 explicitly exculde some dirs from flake8 check 2022-02-08 18:59:13 +05:30
Akash Mahanty 7e7412d9d1 remove deepsource, LGTM is better and has fewer False Postives. 2022-02-08 18:49:44 +05:30
Akash MahantyandGitHub 70c38c5a60 + codecov badge 2022-02-08 17:49:05 +05:30
Akash MahantyandGitHub f8bf9c16f9 Add tests (#149) 
* enable codecov

* fix save_urls_on_file

* increase the limit of CDX to 25000 from 5000. 5X increase.

* added test for the CLI module

* make flake 8 happy

* make mypy happy
 2022-02-08 17:46:59 +05:30
Akash MahantyandGitHub 2bbfee7b2f replace non-ASCII emojis with GitHub hosted equivalent images (#148) 2022-02-08 11:43:32 +05:30
Akash MahantyandGitHub d3a8f343f8 + [eggplants](https://github.com/eggplants) (#143) 2022-02-08 01:41:10 +05:30
Akash MahantyandGitHub 97f8b96411 added docstrings, added some static type hints and also lint. (#141) 
* added docstrings, added some static type hints and also lint.

* added doc strings and changed some internal variable names for more clarity.

* make flake8 happy

* add descriptive docstrings and type hints in waybackpy/cdx_snapshot.py

* remove useless code and add docstrings and also lint using pylint.

* remove unwarented test

* added docstrings, lint using pylint and add a raise on 509 SC

* added docstrings and lint with pylint

* lint

* add doc strings and lint

* add docstrings and lint
 2022-02-07 19:40:37 +05:30
Akash Mahanty a772c22431 explicitly tell pep8speaks that mll is 88. 2022-02-06 21:00:15 +05:30
Akash MahantyandGitHub b79f1c471e Merge pull request #135 from eggplants/fix_cli 
Fix cli.py
 2022-02-05 16:54:36 +05:30
Akash MahantyandGitHub f49d67a411 Merge pull request #136 from eggplants/429_error 
Add TooManyRequestsError
 2022-02-05 11:28:27 +05:30
Akash MahantyandGitHub ad8bd25633 added badge of codacy (#139) 2022-02-05 10:05:17 +05:30
Akash Mahanty 9d9cc3328b add .pep8speaks.yml, override deafult 2022-02-05 00:53:38 +05:30
Akash MahantyandGitHub b69e4dff37 rename params of main in cli.py to avoid using built-ins (#132) 
* rename params of main in cli.py to avoid using built-ins

* Fix Line 32:80: E501 line too long (102 > 79 characters)
 2022-02-05 00:30:35 +05:30
Akash MahantyandGitHub 947647f2e7 Merge pull request #124 from eggplants/fix_save_retry 
Fix save retry mechanism
 2022-02-03 18:01:51 +05:30
Akash Mahanty 5cbdfc040b waybackpy/cli.py : remove duplicate original_string from output_string in cdx 2022-01-30 21:02:25 +05:30
Akash Mahanty 3be6ac01fc created tests/test_cdx_api.py: added tests for cdx_api.py 2022-01-30 20:03:40 +05:30
Akash Mahanty b8b9bc098f tests/test_utils.py: test latest_version_pypi and latest_version_github of waybackpy.utils 2022-01-30 20:02:17 +05:30
Akash Mahanty 946c28eddf waybackpy/cli.py: Added help text, fix bug in the cdx_print parameter and lots of other stuff 
parameter --filters is now --filter

parameter --collapses is now --collapse

added a new --license flag for fetching the license from GitHub repo and printing it.
 2022-01-30 20:00:50 +05:30
Akash Mahanty 004027f73b waybackpy/utils.py : Add a new function(latest_version_github) to fetch the latest release from github api and renamed latest_version to latest_version_pypi as now we have two functions to get the latest release. 2022-01-30 13:28:13 +05:30
Akash MahantyandGitHub e86dd93b29 Delete custom.md 2022-01-30 11:45:51 +05:30
Akash MahantyandGitHub 988568e8f0 Update issue templates 2022-01-30 11:44:30 +05:30
Akash MahantyandGitHub f4c32a44fd Merge pull request #123 from akamhy/add-code-of-conduct-1 
Create CODE_OF_CONDUCT.md
 2022-01-30 11:39:22 +05:30
Akash MahantyandGitHub 7755e6391c Create CODE_OF_CONDUCT.md 2022-01-30 11:39:11 +05:30
Akash Mahanty 9dbe3b3bf4 In waybackpy/wrapper.py set self.timestamp to None on init. 
In older interface(2.x.x) we had timestamp set to none in the constructer, so maybe it should be best to set it to None in the backwards compatiblliy module.)
 2022-01-29 22:12:02 +05:30
Akash MahantyandGitHub e84ba9f2c3 Merge pull request #122 from akamhy/update-readme 
add conda install and related links and tell users that they can copy…
 2022-01-27 00:25:49 +05:30
Akash Mahanty 1250d105b4 update install command for conda and replace the link to conda-forge.org with https://anaconda.org/conda-forge/waybackpy 2022-01-27 00:17:36 +05:30
Akash Mahanty f03b2cb6cb fix formatting of ASCII art 2022-01-26 18:24:24 +05:30
Akash Mahanty 5e0ea023e6 update CLI help text 2022-01-26 16:23:24 +05:30
Akash Mahanty 8dff6f349e add maintainers 2022-01-26 15:45:03 +05:30
Akash Mahanty e04cfdfeaf add conda install and related links and tell users that they can copy text from asciinema.org 2022-01-26 15:40:33 +05:30
Akash MahantyandGitHub 0e2cc8f5ba + asciicast https://asciinema.org/a/464367 
[![asciicast](https://asciinema.org/a/464367.svg)](https://asciinema.org/a/464367)
 2022-01-26 14:51:06 +05:30
Akash Mahanty 9007149fef 3.0.1 -- > 3.0.2, for condaforge staged-recipes issues 2022-01-26 01:54:58 +05:30
Akash Mahanty 8b7603e241 the test is faulty as it fails when we increment the version on dunder version file but did not upstreamed the code to PyPi. 2022-01-26 01:51:24 +05:30
Akash Mahanty 5ea1d3ba4f Replace NON-ASCII character figlet with ASCII character figlet. 2022-01-26 01:46:42 +05:30
Akash Mahanty 4408c5e2ca add snapcraft.yaml 2022-01-25 20:54:09 +05:30
Akash MahantyandGitHub 9afe29a819 Merge pull request #119 from akamhy/akamhy-patch-1 
v3.0.0 --> v3.0.1
 2022-01-25 19:54:01 +05:30
Akash MahantyandGitHub d79b10c74c v3.0.0 --> v3.0.1 2022-01-25 19:52:10 +05:30
Akash Mahanty 32314dc102 Merge branch 'build-test' #118 
Add build test to CI
 see #117
 2022-01-25 14:02:36 +05:30
Akash Mahanty 50e176e2ba .github/workflows/build_test.yml : change python versions from '3.4', '3.8', '3.10' to '3.6', '3.10' as 3.4 not found by GitHub. 2022-01-25 13:56:49 +05:30
Akash Mahanty 4007859c92 Install dependencies for build test in CI : setuptools wheel 2022-01-25 13:35:58 +05:30
Akash Mahanty d8bd6c628d Add build test to CI 2022-01-25 13:30:16 +05:30
Akash MahantyandGitHub 28f6ff8df2 Merge pull request #116 from akamhy/patch-setup-py 
Fix syntax for opening the README.md and __version__.py
 2022-01-25 13:11:33 +05:30
Akash Mahanty 7ac9353f74 Fix syntax for opening the README.md and __version__.py 
For some reason updates made at https://github.com/akamhy/waybackpy/pull/114
are breaking the build using setup, caught while deploying to a cloud service
provider.

The exact error is:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-req-build-n3b9e5pj/setup.py", line 5
  os.path.join(os.path.dirname(__file__), README.md), encoding=utf-8),
                                                                                ^
SyntaxError: invalid syntax
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

See also :
https://github.com/conda-forge/staged-recipes/pull/17634
 2022-01-25 13:05:01 +05:30
Akash MahantyandGitHub 15c7244a22 Merge pull request #115 from akamhy/akamhy-patch-1 
do not use f-strings in setup.py
 2022-01-25 10:42:27 +05:30
Akash MahantyandGitHub 8510210e94 do not use f-strings in setup.py 
These are not supported in <Python 3.6 version of the cpython.
 2022-01-25 10:34:46 +05:30
Akash MahantyandGitHub 552967487e Merge pull request #114 from rafaelrdealmeida/patch-1 
Update setup.py

See also <https://github.com/akamhy/waybackpy/issues/111#issuecomment-1020673814>
 2022-01-25 10:30:34 +05:30
Akash Mahanty 06095202fe BUG FIX : forgot to use the endpoint from the instance and also assign payload to param. Bug caught by the flake8 in the CI tests. 2022-01-24 23:35:48 +05:30
Akash Mahanty 06fc7855bf waybackpy/cdx_api.py : deafult user agent is now DEFAULT_USER_AGENT, get_response now take url and headers as arguments and request url is generated by full_url function. max_tries added as parameter for the WaybackMachineCDXServerAPI class with default value of 3. 2022-01-24 23:20:49 +05:30
Akash Mahanty c49fe971fd update the older deprecation not for Url class, the newer date is now 2025 instead of 2024. 2022-01-24 23:15:59 +05:30
Akash Mahanty d6783d5525 added tests for cdx_utils.py 2022-01-24 23:05:47 +05:30
Akash Mahanty 9262f5da21 improve functions get_total_pages, get_response and lint check_filters, check_collapses and check_match_type 
get_total_pages : default user agent is now DEFAULT_USER_AGENT
                  and now instead of str formatting passing payload
                  as param to full_url to generate the request url
                  also get_response make the request instead of directly
                  using requests.get()

get_response : get_response is now not taking param as keyword arguments
               instead the invoker is supposed to pass the full url which
               may be generated by the full_url function therefore the return_full_url=False,
               is deprecated also.
               Also now closing the session via session.close()
               No need to check 'Exceeded 30 redirects' as save API uses a
               diffrent method.

check_filters : Not assigning to variables the return of match groups
                beacause we wont be using them and the linter picks these
                unused assignments.

check_collapses : Same reason as for check_filters but also removed a foolish
                  test that checks equality with objects that are guaranteed
                  to be same.

check_match_type : Updated the text that of WaybackError
 2022-01-24 22:57:20 +05:30
Akash Mahanty d1a1cf2546 added tests for utils.py at tests/test_utils.py also changed a keyword argument from headers to user_agent for latest_version of utils.py with the usual default vaule. 2022-01-24 17:50:36 +05:30
Akash Mahanty cd8a32ed1f added tests for cdx_snapshot.py at tests/test_cdx_snapshot.py 2022-01-24 16:29:44 +05:30
Akash Mahanty 57512c65ff change test oldest method from google.com to example.com, the oldest on google is for some unknown reason is not very stable. 2022-01-24 16:27:35 +05:30
Akash Mahanty d9ea26e11c added code style black badge 2022-01-24 13:46:31 +05:30
Akash Mahanty 2bea92b348 fix bug with the third matching case of the archive_url_parser, caught while writing more tests fo the save API interface. 2022-01-24 13:31:30 +05:30
Akash Mahanty d506685f68 added some tests for save_api interface 2022-01-23 18:35:54 +05:30
Akash Mahanty 7844d15d99 close the session in save api interface 2022-01-23 18:34:06 +05:30
Akash Mahanty c0252edff2 updated tests for availability_api.py and also added max_tries(default value is 3) with delay (sleep) between successive API calls. The dealy actually improves the performace of the availability_api interface. 2022-01-23 15:05:10 +05:30
Akash Mahanty e7488f3a3e added test badge, rename test to Tests from ubuntu and fix the Incomplete URL substring sanitization(or trying to) 2022-01-23 02:26:53 +05:30
Akash Mahanty aed75ad1db Make modules imprtable as part of a Python package, waybackpy by creating __init__.py file in tests 2022-01-23 02:14:38 +05:30
Akash Mahanty d740959c34 more dev reqs 2022-01-23 02:10:12 +05:30
Akash Mahanty 2d83043ef7 + flake8 in requirements-dev.txt 2022-01-23 02:05:08 +05:30
Akash Mahanty 31b1056217 fix typo in CI 2022-01-23 02:03:30 +05:30
Akash Mahanty 97712b2c1e add CI unit_test.yml 2022-01-23 02:00:15 +05:30
Akash Mahanty a8acc4c4d8 Fix Incomplete URL substring sanitization in the last commit. 2022-01-23 01:42:48 +05:30
Akash Mahanty 1bacd73002 created pytest.ini, added test for waybackpy/availability_api.py, new exceptions all of which inherit from the main WaybackError and created requirements-dev.txt 2022-01-23 01:29:07 +05:30
Akash Mahanty 79901ba968 updated README.md 2022-01-22 03:08:26 +05:30
Akash Mahanty df64e839d7 added trove classifiers for python 3.10 2022-01-22 00:57:10 +05:30
Akash Mahanty 405e9a2a79 waybackpy/save_api.py : Added doc strings and also lint with black. 2022-01-22 00:41:10 +05:30
Akash Mahanty db551abbf6 lint waybackpy/cdx_api.py and added some doc strings 2022-01-22 00:11:35 +05:30
Akash Mahanty d13dd4db1a added notice on waybackpy/wrapper.py that the Url class will cease to exist after 2024-01-01 and also removed unused imports. 2022-01-21 23:14:20 +05:30
Akash Mahanty d3bb8337a1 make setup.py smarter, now no need to update the URL again and also added more keywords. And in __version__.py updated the __author__ 2022-01-21 23:01:09 +05:30
Akash Mahanty fd5e85420c waybackpy/availability_api.py : removed unused imports, added doc strings, removed redundant function. 2022-01-21 22:47:44 +05:30
Akash MahantyandGitHub 5c685ef5d7 upload logo and make p path not text 
I was dumb to forget to convert the p to path.
 2022-01-21 21:11:42 +05:30