Benjamin_Loison/waybackpy
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity
478 Commits 1 Branch 37 Tags
87fb5ecd58a6bf22bc32e1a0a9697176ff9f5d65
Commit Graph

211 Commits

Author SHA1 Message Date
Akash Mahanty
16b9bdd7f9 output the file name if known_url and file flag are passed. 2022-01-18 20:14:44 +05:30
Akash Mahanty
7adc01bff2 implement known_urls for cli from the newer interface. Although use of CDX is recommended but backward-compatibility matters. 2022-01-18 20:07:12 +05:30
Akash Mahanty
cc3628ae18 define __str__ for objects of WaybackMachineAvailabilityAPI class, the check for self.JSON ensures that the API was atleast called. 2022-01-16 22:28:12 +05:30
Akash Mahanty
1d751b942b invoke json, was a bad idea removing it the earlier commit as the end user should not have to call it 2022-01-16 22:15:25 +05:30
Akash Mahanty
261a867a21 near() method of WaybackMachineAvailabilityAPI return self to preserve past behaviour 2022-01-16 21:53:54 +05:30
Akash Mahanty
2e487e88d3 define __len__ on Url objects, if any method not used prior to len op then default to len of oldest archive. 2022-01-16 21:29:43 +05:30
Akash Mahanty
c8d0ad493a defined __str__ for Url objects, print func should print the url. 2022-01-16 21:22:43 +05:30
Akash Mahanty
4e68cd5743 Create separate module for the 3 different APIs also CDX is now CLI supported. 2022-01-02 14:14:45 +05:30
akamhy
a7b805292d changes made for v2.4.4 (update download_url) (#100) 
* v2.4.4 (update download_url)

* v2.4.4 (update __version__)

* +1

add jonasjancarik
 2021-09-03 11:28:26 +05:30
Jonáš Jančařík
6dc6124dc4 Raise error on a 509 response (too many sessions) (#99) 
* Raise error on a 509 response (too many sessions)

When the response code is 509, raise an error with an explanation (based on the actual error message contained in the response HTML).

* Raise error on a 509 response (too many sessions) - linting
 2021-09-03 08:04:36 +05:30
Jens Finkhaeuser
5a7fc7d568 Fix typo (#95) 2021-04-13 16:58:34 +05:30
Akash Mahanty
5a9c861cad v2.4.3 (#94) 
* 2.4.3

* 2.4.3
 2021-04-02 10:41:59 +05:30
Akash Mahanty
dd1917c77e added RedirectSaveError - for failed saves if the URL is a permanent … (#93) 
* added RedirectSaveError - for failed saves if the URL is a permanent redirect.

* check if url is redirect before throwing exceptions, res.url is the redirect url if redirected at all

* update tests and cli errors
 2021-04-02 10:38:17 +05:30
Akash Mahanty
db8f902cff Add doc strings (#90) 
* Added some docstrings in utils.py

* renamed some func/meth to better names and added doc strings + lint

* added more docstrings

* more docstrings

* improve docstrings

* docstrings

* added more docstrings, lint

* fix import error
 2021-01-26 11:56:03 +05:30
Akash Mahanty
88cda94c0b v2.4.2 (#89) 
* v2.4.2

* v2.4.2
 2021-01-24 17:03:35 +05:30
Akash Mahanty
09290f88d1 fix one more error 2021-01-24 16:58:53 +05:30
Akash Mahanty
e5835091c9 import re 2021-01-24 16:56:59 +05:30
Akash Mahanty
7312ed1f4f set cached_save to True if archive older than 3 mins. 2021-01-24 16:53:36 +05:30
Akash Mahanty
36b936820b known urls now yileds, more reliable. And save the file in chucks wrt to response. --file arg can be used to create output file, if --file not used no output will be saved in any file. (#88) 2021-01-24 16:11:39 +05:30
Akash Mahanty
edc2f63d93 Output valid JSON, dumps python dict. Make JSON valid. 2021-01-23 20:43:52 +05:30
Akash Mahanty
ffe0810b12 flag to check if the archive saved is 30 mins older or not 2021-01-16 12:06:08 +05:30
Akash Mahanty
40233eb115 improve code quality, remove unused imports, use system randomness etc 2021-01-16 11:35:13 +05:30
Akash Mahanty
d549d31421 improve save method, now we know that 302 errors indicates that wayback machine is archiving the URL and hasn't yet archived. We construct an artifical archive with the current UTC time and check for HTTP status code 20* or 30*. If we verify the archival, we return the artifical archive. The artificial archive will automatically point to the new archive or in best case will be the new archive after some time. 2021-01-16 10:47:43 +05:30
Akash Mahanty
712471176b better error messages(str), check latest version before asking for an upgrade and rm alive checking 2021-01-15 16:47:26 +05:30
Akash Mahanty
dcd7b03302 getting rid of c style str formatting, now using .format 2021-01-14 19:30:07 +05:30
Akash Mahanty
76205d9cf6 backoff_factor=2 for save, incr success by 25% 2021-01-13 10:13:16 +05:30
Akash Mahanty
7bb01df846 v2.4.1 2021-01-12 10:18:09 +05:30
Akash Mahanty
6142e0b353 get should retrive the last fetched archive by default 2021-01-12 10:07:14 +05:30
Akash Mahanty
a65990aee3 don't use pagination API if total pages <= 2 2021-01-12 09:46:07 +05:30
Akash Mahanty
eabf4dc046 don't fetch more pages if >=2 pages are empty 2021-01-11 22:43:14 +05:30
Akash Mahanty
5a7bd73565 support unix ts as an arg in near 2021-01-11 19:53:37 +05:30
Akash Mahanty
4693dbf9c1 change str repr of cdxsnapshot to cdx line 2021-01-11 09:34:37 +05:30
Akash Mahanty
f4f2e51315 V2.4.0 (#62) 
* v 2.4.0

* v 2.4.0
 2021-01-10 11:53:45 +05:30
Akash Mahanty
d6b7df6837 no need to de-duplicate as we are collapsing the results by urlkey 
Same urls aren't recieved
 2021-01-10 11:36:46 +05:30
Akash Mahanty
dafba5d0cb collapses=["urlkey"] for known urls 2021-01-10 11:34:06 +05:30
Akash Mahanty
6c71dfbe41 use cdx matchtype for domain and host 2021-01-10 11:10:49 +05:30
Akash Mahanty
a6470b1036 not passing dict to cdxsnapshot 2021-01-10 10:40:32 +05:30
Akash Mahanty
625ed63482 remove asserts stmnts 2021-01-10 03:05:48 +05:30
Akash Mahanty
a03813315f full cdx api support 2021-01-10 02:23:53 +05:30
Akash Mahanty
a2550f17d7 retries support for get requests 2021-01-06 01:58:38 +05:30
Akash Mahanty
15ef5816db Always cast url to string, avoid passing waybackpy objects to _get_response 2021-01-05 19:46:17 +05:30
Akash Mahanty
93b52bd0fe FIX : don't use self.user_agent if user_agent passed in get() 2021-01-05 19:31:27 +05:30
Akash Mahanty
d98c4f32ad v2.3.3 2021-01-05 01:48:54 +05:30
Akash Mahanty
e0a4b007d5 improve docs 2021-01-05 01:46:12 +05:30
Akash Mahanty
1882862992 now using cdx Pagination API 2021-01-04 20:46:54 +05:30
Akash Mahanty
0c6107e675 increase coverage 2021-01-04 01:54:40 +05:30
Akash Mahanty
5dec4927cd refactoring, try to code complexity 2021-01-04 00:14:38 +05:30
Akash Mahanty
62e5217b9e reduce code complexity: refactoring, less flow breaking structures 2021-01-03 19:38:25 +05:30
Akash Mahanty
9823c809e9 Added doc strings in wrapper.py, documenting code and improving docs. 2021-01-03 17:11:32 +05:30
Akash Mahanty
db5737a857 JSON is now available for near and other other methods that call it 2021-01-02 18:52:46 +05:30