Benjamin_Loison/waybackpy
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity
432 Commits 1 Branch 37 Tags
e86dd93b297d67301fd4b0d0360e02f3ba0a583d
Commit Graph

185 Commits

Author SHA1 Message Date
Akash Mahanty
9dbe3b3bf4 In waybackpy/wrapper.py set self.timestamp to None on init. 
In older interface(2.x.x) we had timestamp set to none in the constructer, so maybe it should be best to set it to None in the backwards compatiblliy module.)
 2022-01-29 22:12:02 +05:30
Akash Mahanty
f03b2cb6cb fix formatting of ASCII art 2022-01-26 18:24:24 +05:30
Akash Mahanty
5e0ea023e6 update CLI help text 2022-01-26 16:23:24 +05:30
Akash Mahanty
9007149fef 3.0.1 -- > 3.0.2, for condaforge staged-recipes issues 2022-01-26 01:54:58 +05:30
Akash Mahanty
5ea1d3ba4f Replace NON-ASCII character figlet with ASCII character figlet. 2022-01-26 01:46:42 +05:30
Akash Mahanty
d79b10c74c v3.0.0 --> v3.0.1 2022-01-25 19:52:10 +05:30
Akash Mahanty
06095202fe BUG FIX : forgot to use the endpoint from the instance and also assign payload to param. Bug caught by the flake8 in the CI tests. 2022-01-24 23:35:48 +05:30
Akash Mahanty
06fc7855bf waybackpy/cdx_api.py : deafult user agent is now DEFAULT_USER_AGENT, get_response now take url and headers as arguments and request url is generated by full_url function. max_tries added as parameter for the WaybackMachineCDXServerAPI class with default value of 3. 2022-01-24 23:20:49 +05:30
Akash Mahanty
c49fe971fd update the older deprecation not for Url class, the newer date is now 2025 instead of 2024. 2022-01-24 23:15:59 +05:30
Akash Mahanty
9262f5da21 improve functions get_total_pages, get_response and lint check_filters, check_collapses and check_match_type 
get_total_pages : default user agent is now DEFAULT_USER_AGENT
                  and now instead of str formatting passing payload
                  as param to full_url to generate the request url
                  also get_response make the request instead of directly
                  using requests.get()

get_response : get_response is now not taking param as keyword arguments
               instead the invoker is supposed to pass the full url which
               may be generated by the full_url function therefore the return_full_url=False,
               is deprecated also.
               Also now closing the session via session.close()
               No need to check 'Exceeded 30 redirects' as save API uses a
               diffrent method.

check_filters : Not assigning to variables the return of match groups
                beacause we wont be using them and the linter picks these
                unused assignments.

check_collapses : Same reason as for check_filters but also removed a foolish
                  test that checks equality with objects that are guaranteed
                  to be same.

check_match_type : Updated the text that of WaybackError
 2022-01-24 22:57:20 +05:30
Akash Mahanty
d1a1cf2546 added tests for utils.py at tests/test_utils.py also changed a keyword argument from headers to user_agent for latest_version of utils.py with the usual default vaule. 2022-01-24 17:50:36 +05:30
Akash Mahanty
cd8a32ed1f added tests for cdx_snapshot.py at tests/test_cdx_snapshot.py 2022-01-24 16:29:44 +05:30
Akash Mahanty
2bea92b348 fix bug with the third matching case of the archive_url_parser, caught while writing more tests fo the save API interface. 2022-01-24 13:31:30 +05:30
Akash Mahanty
7844d15d99 close the session in save api interface 2022-01-23 18:34:06 +05:30
Akash Mahanty
c0252edff2 updated tests for availability_api.py and also added max_tries(default value is 3) with delay (sleep) between successive API calls. The dealy actually improves the performace of the availability_api interface. 2022-01-23 15:05:10 +05:30
Akash Mahanty
1bacd73002 created pytest.ini, added test for waybackpy/availability_api.py, new exceptions all of which inherit from the main WaybackError and created requirements-dev.txt 2022-01-23 01:29:07 +05:30
Akash Mahanty
79901ba968 updated README.md 2022-01-22 03:08:26 +05:30
Akash Mahanty
405e9a2a79 waybackpy/save_api.py : Added doc strings and also lint with black. 2022-01-22 00:41:10 +05:30
Akash Mahanty
db551abbf6 lint waybackpy/cdx_api.py and added some doc strings 2022-01-22 00:11:35 +05:30
Akash Mahanty
d13dd4db1a added notice on waybackpy/wrapper.py that the Url class will cease to exist after 2024-01-01 and also removed unused imports. 2022-01-21 23:14:20 +05:30
Akash Mahanty
d3bb8337a1 make setup.py smarter, now no need to update the URL again and also added more keywords. And in __version__.py updated the __author__ 2022-01-21 23:01:09 +05:30
Akash Mahanty
fd5e85420c waybackpy/availability_api.py : removed unused imports, added doc strings, removed redundant function. 2022-01-21 22:47:44 +05:30
Akash Mahanty
5e9fdb40ce escape '.' before 'archive.org' 
escape '.' before 'archive.org' on line 88 so it does not match more hosts than expected.
 2022-01-21 19:51:08 +05:30
Akash Mahanty
66e16d6d89 define __repr__ for the Availability API class 2022-01-18 20:34:21 +05:30
Akash Mahanty
16b9bdd7f9 output the file name if known_url and file flag are passed. 2022-01-18 20:14:44 +05:30
Akash Mahanty
7adc01bff2 implement known_urls for cli from the newer interface. Although use of CDX is recommended but backward-compatibility matters. 2022-01-18 20:07:12 +05:30
Akash Mahanty
cc3628ae18 define __str__ for objects of WaybackMachineAvailabilityAPI class, the check for self.JSON ensures that the API was atleast called. 2022-01-16 22:28:12 +05:30
Akash Mahanty
1d751b942b invoke json, was a bad idea removing it the earlier commit as the end user should not have to call it 2022-01-16 22:15:25 +05:30
Akash Mahanty
261a867a21 near() method of WaybackMachineAvailabilityAPI return self to preserve past behaviour 2022-01-16 21:53:54 +05:30
Akash Mahanty
2e487e88d3 define __len__ on Url objects, if any method not used prior to len op then default to len of oldest archive. 2022-01-16 21:29:43 +05:30
Akash Mahanty
c8d0ad493a defined __str__ for Url objects, print func should print the url. 2022-01-16 21:22:43 +05:30
Akash Mahanty
4e68cd5743 Create separate module for the 3 different APIs also CDX is now CLI supported. 2022-01-02 14:14:45 +05:30
akamhy
a7b805292d changes made for v2.4.4 (update download_url) (#100) 
* v2.4.4 (update download_url)

* v2.4.4 (update __version__)

* +1

add jonasjancarik
 2021-09-03 11:28:26 +05:30
Jonáš Jančařík
6dc6124dc4 Raise error on a 509 response (too many sessions) (#99) 
* Raise error on a 509 response (too many sessions)

When the response code is 509, raise an error with an explanation (based on the actual error message contained in the response HTML).

* Raise error on a 509 response (too many sessions) - linting
 2021-09-03 08:04:36 +05:30
Jens Finkhaeuser
5a7fc7d568 Fix typo (#95) 2021-04-13 16:58:34 +05:30
Akash Mahanty
5a9c861cad v2.4.3 (#94) 
* 2.4.3

* 2.4.3
 2021-04-02 10:41:59 +05:30
Akash Mahanty
dd1917c77e added RedirectSaveError - for failed saves if the URL is a permanent … (#93) 
* added RedirectSaveError - for failed saves if the URL is a permanent redirect.

* check if url is redirect before throwing exceptions, res.url is the redirect url if redirected at all

* update tests and cli errors
 2021-04-02 10:38:17 +05:30
Akash Mahanty
db8f902cff Add doc strings (#90) 
* Added some docstrings in utils.py

* renamed some func/meth to better names and added doc strings + lint

* added more docstrings

* more docstrings

* improve docstrings

* docstrings

* added more docstrings, lint

* fix import error
 2021-01-26 11:56:03 +05:30
Akash Mahanty
88cda94c0b v2.4.2 (#89) 
* v2.4.2

* v2.4.2
 2021-01-24 17:03:35 +05:30
Akash Mahanty
09290f88d1 fix one more error 2021-01-24 16:58:53 +05:30
Akash Mahanty
e5835091c9 import re 2021-01-24 16:56:59 +05:30
Akash Mahanty
7312ed1f4f set cached_save to True if archive older than 3 mins. 2021-01-24 16:53:36 +05:30
Akash Mahanty
36b936820b known urls now yileds, more reliable. And save the file in chucks wrt to response. --file arg can be used to create output file, if --file not used no output will be saved in any file. (#88) 2021-01-24 16:11:39 +05:30
Akash Mahanty
edc2f63d93 Output valid JSON, dumps python dict. Make JSON valid. 2021-01-23 20:43:52 +05:30
Akash Mahanty
ffe0810b12 flag to check if the archive saved is 30 mins older or not 2021-01-16 12:06:08 +05:30
Akash Mahanty
40233eb115 improve code quality, remove unused imports, use system randomness etc 2021-01-16 11:35:13 +05:30
Akash Mahanty
d549d31421 improve save method, now we know that 302 errors indicates that wayback machine is archiving the URL and hasn't yet archived. We construct an artifical archive with the current UTC time and check for HTTP status code 20* or 30*. If we verify the archival, we return the artifical archive. The artificial archive will automatically point to the new archive or in best case will be the new archive after some time. 2021-01-16 10:47:43 +05:30
Akash Mahanty
712471176b better error messages(str), check latest version before asking for an upgrade and rm alive checking 2021-01-15 16:47:26 +05:30
Akash Mahanty
dcd7b03302 getting rid of c style str formatting, now using .format 2021-01-14 19:30:07 +05:30
Akash Mahanty
76205d9cf6 backoff_factor=2 for save, incr success by 25% 2021-01-13 10:13:16 +05:30