Benjamin_Loison/waybackpy
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity
390 Commits 1 Branch 37 Tags
e7488f3a3e40442f6a670de1c9aeef5691328685
Commit Graph
99 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Akash Mahanty e7488f3a3e added test badge, rename test to Tests from ubuntu and fix the Incomplete URL substring sanitization(or trying to) 2022-01-23 02:26:53 +05:30
Akash Mahanty aed75ad1db Make modules imprtable as part of a Python package, waybackpy by creating __init__.py file in tests 2022-01-23 02:14:38 +05:30
Akash Mahanty d740959c34 more dev reqs 2022-01-23 02:10:12 +05:30
Akash Mahanty 2d83043ef7 + flake8 in requirements-dev.txt 2022-01-23 02:05:08 +05:30
Akash Mahanty 31b1056217 fix typo in CI 2022-01-23 02:03:30 +05:30
Akash Mahanty 97712b2c1e add CI unit_test.yml 2022-01-23 02:00:15 +05:30
Akash Mahanty a8acc4c4d8 Fix Incomplete URL substring sanitization in the last commit. 2022-01-23 01:42:48 +05:30
Akash Mahanty 1bacd73002 created pytest.ini, added test for waybackpy/availability_api.py, new exceptions all of which inherit from the main WaybackError and created requirements-dev.txt 2022-01-23 01:29:07 +05:30
Akash Mahanty 79901ba968 updated README.md 2022-01-22 03:08:26 +05:30
Akash Mahanty df64e839d7 added trove classifiers for python 3.10 2022-01-22 00:57:10 +05:30
Akash Mahanty 405e9a2a79 waybackpy/save_api.py : Added doc strings and also lint with black. 2022-01-22 00:41:10 +05:30
Akash Mahanty db551abbf6 lint waybackpy/cdx_api.py and added some doc strings 2022-01-22 00:11:35 +05:30
Akash Mahanty d13dd4db1a added notice on waybackpy/wrapper.py that the Url class will cease to exist after 2024-01-01 and also removed unused imports. 2022-01-21 23:14:20 +05:30
Akash Mahanty d3bb8337a1 make setup.py smarter, now no need to update the URL again and also added more keywords. And in __version__.py updated the __author__ 2022-01-21 23:01:09 +05:30
Akash Mahanty fd5e85420c waybackpy/availability_api.py : removed unused imports, added doc strings, removed redundant function. 2022-01-21 22:47:44 +05:30
Akash MahantyandGitHub 5c685ef5d7 upload logo and make p path not text 
I was dumb to forget to convert the p to path.
 2022-01-21 21:11:42 +05:30
Akash MahantyandGitHub 6a3d96b453 Logo (#113) 
* Create logo.txt

* Delete waybackpy_logo.svg

* Add files via upload

* Delete logo.txt
 2022-01-21 21:02:38 +05:30
Akash MahantyandGitHub afe1b15a5f Add files via upload 2022-01-21 20:58:53 +05:30
Akash MahantyandGitHub 4fd9d142e7 Merge pull request #112 from akamhy/fix 
escape '.' before 'archive.org'
 2022-01-21 19:52:55 +05:30
Akash MahantyandGitHub 5e9fdb40ce escape '.' before 'archive.org' 
escape '.' before 'archive.org' on line 88 so it does not match more hosts than expected.
 2022-01-21 19:51:08 +05:30
Akash MahantyandGitHub fa72098270 _get_response is not used anymore 
- datashaman (<https://stackoverflow.com/users/401467/datashaman>) for <https://stackoverflow.com/a/35504626>. _get_response is based on this amazing answer.
 2022-01-21 19:43:35 +05:30
Akash MahantyandGitHub d18f955044 date year range 2020-2022 2022-01-21 11:55:42 +05:30
Akash MahantyandGitHub 9c340d6967 Create codeql-analysis.yml 2022-01-21 11:12:59 +05:30
Akash MahantyandGitHub 78d0e0c126 Update README.md 2022-01-21 09:54:04 +05:30
Akash MahantyandGitHub 564101e6f5 🐳 for docker image 2022-01-21 01:23:05 +05:30
Akash MahantyandGitHub de5a3e1561 improve usage code 2022-01-18 21:18:17 +05:30
Akash MahantyandGitHub 52e46fecc2 more usage example 2022-01-18 20:58:39 +05:30
Akash MahantyandGitHub 3b6415abc7 updating examples 2022-01-18 20:44:47 +05:30
Akash Mahanty 66e16d6d89 define __repr__ for the Availability API class 2022-01-18 20:34:21 +05:30
Akash Mahanty 16b9bdd7f9 output the file name if known_url and file flag are passed. 2022-01-18 20:14:44 +05:30
Akash Mahanty 7adc01bff2 implement known_urls for cli from the newer interface. Although use of CDX is recommended but backward-compatibility matters. 2022-01-18 20:07:12 +05:30
Akash MahantyandGitHub 9bbd056268 Update README.md 2022-01-17 02:15:38 +05:30
Akash MahantyandGitHub 2ab44391cf close #107, added link to SecSI/Docker image 2022-01-16 23:01:31 +05:30
Akash Mahanty cc3628ae18 define __str__ for objects of WaybackMachineAvailabilityAPI class, the check for self.JSON ensures that the API was atleast called. 2022-01-16 22:28:12 +05:30
Akash Mahanty 1d751b942b invoke json, was a bad idea removing it the earlier commit as the end user should not have to call it 2022-01-16 22:15:25 +05:30
Akash Mahanty 261a867a21 near() method of WaybackMachineAvailabilityAPI return self to preserve past behaviour 2022-01-16 21:53:54 +05:30
Akash Mahanty 2e487e88d3 define __len__ on Url objects, if any method not used prior to len op then default to len of oldest archive. 2022-01-16 21:29:43 +05:30
Akash Mahanty c8d0ad493a defined __str__ for Url objects, print func should print the url. 2022-01-16 21:22:43 +05:30
Akash MahantyandGitHub ce869177fd Merge pull request #103 from akamhy/whitesource/configure 
Configure WhiteSource Bolt for GitHub
 2022-01-02 16:04:15 +05:30
Akash Mahanty 4e68cd5743 Create separate module for the 3 different APIs also CDX is now CLI supported. 2022-01-02 14:14:45 +05:30
akamhyandGitHub a7b805292d changes made for v2.4.4 (update download_url) (#100) 
* v2.4.4 (update download_url)

* v2.4.4 (update __version__)

* +1

add jonasjancarik
 2021-09-03 11:28:26 +05:30
Akash MahantyandGitHub 5a9c861cad v2.4.3 (#94) 
* 2.4.3

* 2.4.3
 2021-04-02 10:41:59 +05:30
Akash MahantyandGitHub dd1917c77e added RedirectSaveError - for failed saves if the URL is a permanent … (#93) 
* added RedirectSaveError - for failed saves if the URL is a permanent redirect.

* check if url is redirect before throwing exceptions, res.url is the redirect url if redirected at all

* update tests and cli errors
 2021-04-02 10:38:17 +05:30
Akash MahantyandGitHub db8f902cff Add doc strings (#90) 
* Added some docstrings in utils.py

* renamed some func/meth to better names and added doc strings + lint

* added more docstrings

* more docstrings

* improve docstrings

* docstrings

* added more docstrings, lint

* fix import error
 2021-01-26 11:56:03 +05:30
Akash MahantyandGitHub 88cda94c0b v2.4.2 (#89) 
* v2.4.2

* v2.4.2
 2021-01-24 17:03:35 +05:30
Akash Mahanty 09290f88d1 fix one more error 2021-01-24 16:58:53 +05:30
Akash Mahanty e5835091c9 import re 2021-01-24 16:56:59 +05:30
Akash Mahanty 7312ed1f4f set cached_save to True if archive older than 3 mins. 2021-01-24 16:53:36 +05:30
Akash MahantyandGitHub 6ae8f843d3 add --file to --known_urls 2021-01-24 16:15:11 +05:30
Akash MahantyandGitHub 36b936820b known urls now yileds, more reliable. And save the file in chucks wrt to response. --file arg can be used to create output file, if --file not used no output will be saved in any file. (#88) 2021-01-24 16:11:39 +05:30
Akash Mahanty a3bc6aad2b too much API usage by duplicate tests was causing too much tests failure 2021-01-23 21:08:21 +05:30
Akash Mahanty edc2f63d93 Output valid JSON, dumps python dict. Make JSON valid. 2021-01-23 20:43:52 +05:30
Akash Mahanty ffe0810b12 flag to check if the archive saved is 30 mins older or not 2021-01-16 12:06:08 +05:30
Akash Mahanty 40233eb115 improve code quality, remove unused imports, use system randomness etc 2021-01-16 11:35:13 +05:30
Akash Mahanty d549d31421 improve save method, now we know that 302 errors indicates that wayback machine is archiving the URL and hasn't yet archived. We construct an artifical archive with the current UTC time and check for HTTP status code 20* or 30*. If we verify the archival, we return the artifical archive. The artificial archive will automatically point to the new archive or in best case will be the new archive after some time. 2021-01-16 10:47:43 +05:30
Akash Mahanty 0725163af8 mimify the logo, remove ugly old logos 2021-01-15 18:14:48 +05:30
Akash Mahanty 712471176b better error messages(str), check latest version before asking for an upgrade and rm alive checking 2021-01-15 16:47:26 +05:30
Akash Mahanty dcd7b03302 getting rid of c style str formatting, now using .format 2021-01-14 19:30:07 +05:30
Akash Mahanty 76205d9cf6 backoff_factor=2 for save, incr success by 25% 2021-01-13 10:13:16 +05:30
Akash MahantyandGitHub ec0a0d04cc + dequeued0 
dequeued0 (https://github.com/dequeued0) for reporting bugs and useful feature requests.
 2021-01-12 10:52:41 +05:30
Akash Mahanty 7bb01df846 v2.4.1 2021-01-12 10:18:09 +05:30
Akash Mahanty 6142e0b353 get should retrive the last fetched archive by default 2021-01-12 10:07:14 +05:30
Akash Mahanty a65990aee3 don't use pagination API if total pages <= 2 2021-01-12 09:46:07 +05:30
Akash MahantyandGitHub 259a024eb1 joke? they changed their robots.txt 2021-01-11 23:17:01 +05:30
Akash MahantyandGitHub 91402792e6 + Supported Features 
tell what the package can do, many users probably do not read the full usage.
 2021-01-11 23:01:18 +05:30
Akash Mahanty eabf4dc046 don't fetch more pages if >=2 pages are empty 2021-01-11 22:43:14 +05:30
Akash Mahanty 5a7bd73565 support unix ts as an arg in near 2021-01-11 19:53:37 +05:30
Akash Mahanty 4693dbf9c1 change str repr of cdxsnapshot to cdx line 2021-01-11 09:34:37 +05:30
Akash MahantyandGitHub f4f2e51315 V2.4.0 (#62) 
* v 2.4.0

* v 2.4.0
 2021-01-10 11:53:45 +05:30
Akash MahantyandGitHub d6b7df6837 no need to de-duplicate as we are collapsing the results by urlkey 
Same urls aren't recieved
 2021-01-10 11:36:46 +05:30
Akash MahantyandGitHub dafba5d0cb collapses=["urlkey"] for known urls 2021-01-10 11:34:06 +05:30
Akash Mahanty 6c71dfbe41 use cdx matchtype for domain and host 2021-01-10 11:10:49 +05:30
Akash Mahanty a6470b1036 not passing dict to cdxsnapshot 2021-01-10 10:40:32 +05:30
Akash Mahanty 04cda4558e fix test 2021-01-10 03:18:09 +05:30
Akash Mahanty 625ed63482 remove asserts stmnts 2021-01-10 03:05:48 +05:30
Akash Mahanty a03813315f full cdx api support 2021-01-10 02:23:53 +05:30
Akash Mahanty a2550f17d7 retries support for get requests 2021-01-06 01:58:38 +05:30
Akash MahantyandGitHub 15ef5816db Always cast url to string, avoid passing waybackpy objects to _get_response 2021-01-05 19:46:17 +05:30
Akash MahantyandGitHub 93b52bd0fe FIX : don't use self.user_agent if user_agent passed in get() 2021-01-05 19:31:27 +05:30
Akash MahantyandGitHub 28ff877081 Update README.md 2021-01-05 19:08:35 +05:30
Akash MahantyandGitHub 3e3ecff9df l2 heading and lint 2021-01-05 01:59:29 +05:30
Akash MahantyandGitHub ce64135ba8 ce 2021-01-05 01:52:35 +05:30
Akash MahantyandGitHub 2af6580ffb docs link 2021-01-05 01:51:53 +05:30
Akash MahantyandGitHub 8a3c515176 v2.3.3 2021-01-05 01:49:26 +05:30
Akash MahantyandGitHub d98c4f32ad v2.3.3 2021-01-05 01:48:54 +05:30
Akash Mahanty e0a4b007d5 improve docs 2021-01-05 01:46:12 +05:30
Akash MahantyandGitHub 6fb6b2deee Update readme + new file CONTRIBUTORS.md (#59) 
* remove some badges

* remove made with python button, obvious

* - maintained badge, we already have latest commit badge

- [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/akamhy/waybackpy/graphs/commit-activity)

* re arranged order of badges

* a bit more re odering

* - release badge

* - license section

* center h1

* try once more'

* removed the TOC

* move the hr

* Update README.md

* + hr

* h1 --> h2

* remove tests and pacakging info from here to docs/wiki

* Update README.md

* example inspired by psf/requests

* CLI tool example gist

* Update README.md

* Update README.md

* + license

* Update README.md

* authors list

* Update CONTRIBUTORS.md

* fix code

* Update README.md

* Update README.md

* center the button
 2021-01-05 00:30:07 +05:30
Akash Mahanty 1882862992 now using cdx Pagination API 2021-01-04 20:46:54 +05:30
Akash Mahanty 0c6107e675 increase coverage 2021-01-04 01:54:40 +05:30
Akash Mahanty bd079978bf inc coverage 2021-01-04 00:44:55 +05:30
Akash Mahanty 5dec4927cd refactoring, try to code complexity 2021-01-04 00:14:38 +05:30
Akash Mahanty 62e5217b9e reduce code complexity: refactoring, less flow breaking structures 2021-01-03 19:38:25 +05:30
Akash Mahanty 9823c809e9 Added doc strings in wrapper.py, documenting code and improving docs. 2021-01-03 17:11:32 +05:30
Akash Mahanty db5737a857 JSON is now available for near and other other methods that call it 2021-01-02 18:52:46 +05:30
Akash MahantyandGitHub ca0821a466 Wiki docs (#58) 
* move docs to wiki

* Update README.md

* Update setup.py
 2021-01-02 12:20:43 +05:30
Akash MahantyandGitHub bb4dbc7d3c rm url = obj.url 2021-01-02 11:19:09 +05:30
Akash MahantyandGitHub 7c7fd75376 No need to fetch archive_url and timestamp from availability API on init (#55) 
* No need to fetch archive_url and timestamp from availability API on init. 

Not useful if all I want is to archive a page

* Update test_wrapper.py

* Update wrapper.py

* Update test_wrapper.py

* Update wrapper.py

* Update cli.py

* Update wrapper.py

* Update __version__.py

* Update __version__.py

* Update __version__.py

* Update __version__.py

* Update setup.py

* Update README.md
 2021-01-02 11:10:23 +05:30
Akash MahantyandGitHub 0b71433667 v2.3.1 (#54) 
* 2.3.1

* 2.3.1
 2021-01-01 19:15:23 +05:30
Akash MahantyandGitHub 1b499a7594 removed JSON from init, this was resulting in too much unnecessary taffic. Some users who are thousands of URLs were blocked by IA (#53) 
closes #52
 2021-01-01 16:38:57 +05:30