misc-mirrors/article_scraper
1
0
Fork 0
mirror of https://gitlab.com/news-flash/article_scraper.git synced 2025-07-08 00:19:59 +02:00
Code Releases Activity
300 commits 6 branches 13 tags 11 MiB
Commit graph

300 commits

This branch
This branch
All branches
Author SHA1 Message Date
Jan Lukas Gernert
1fd7173eac update for newer deps 2020-04-28 02:51:30 +02:00
Jan Lukas Gernert
1fbce6413d Merge branch 'master' of gitlab.com:news-flash/article_scraper 2020-04-28 02:34:24 +02:00
Jan Lukas Gernert
f6d021b67b first release 2020-04-28 02:33:25 +02:00
Jan Lukas Gernert
d2960d8539 require client for parsing 2020-02-10 18:01:35 +01:00
Jan Lukas Gernert
a7c247549a remve unused crate 2020-02-06 21:08:58 +01:00
Jan Lukas Gernert
1ecc0fc4b4 option to set custom reqwest client 2020-02-03 17:46:54 +01:00
Jan Lukas Gernert
71055eed1c fix corrupt filename 2020-01-27 17:32:17 +01:00
Jan Lukas Gernert
98348b7e59 tmp: dont strip scripts 2020-01-27 16:47:13 +01:00
Jan Lukas Gernert
23514aff9e less dramatic logging 2020-01-27 02:03:06 +01:00
Jan Lukas Gernert
afe661fe6c only go for single page link if xpath res isn't empty 2020-01-27 01:54:37 +01:00
Jan Lukas Gernert
e58acf828c improve logging clearity 2020-01-27 01:48:54 +01:00
Jan Lukas Gernert
c720dbc299 fixup 2020-01-27 01:35:15 +01:00
Jan Lukas Gernert
b272c99911 fix missing '/' in url completion 2020-01-27 01:21:21 +01:00
Jan Lukas Gernert
f570873aba load config files in background thread 2020-01-26 21:44:26 +01:00
Jan Lukas Gernert
2cac8a2678 got back to stable libxml 2020-01-26 17:34:47 +01:00
Jan Lukas Gernert
8025e8f004 Merge branch 'async' into 'master' 
Async

See merge request news-flash/article_scraper!3
 2020-01-19 21:15:13 +00:00
Jan Lukas Gernert
d9c7ef1471 Merge branch 'master' into 'async' 
# Conflicts:
#   Cargo.toml
#   src/images/mod.rs
#   src/lib.rs
 2020-01-19 21:15:08 +00:00
Jan Lukas Gernert
d843809437 update reqwest to stable 2020-01-18 19:06:53 +01:00
Jan Lukas Gernert
9e995122c4 only strip topmost nodes in tree branches 2019-12-19 17:36:48 +01:00
Jan Lukas Gernert
b032ec99bc Merge branch 'async' into 'master' 
Async

See merge request news-flash/article_scraper!2
 2019-12-16 11:37:00 +00:00
Jan Lukas Gernert
9c35fb9fa8 Async 2019-12-16 11:36:59 +00:00
Jan Lukas Gernert
26346839f2 remove prints 2019-11-19 19:28:49 +01:00
Jan Lukas Gernert
edfbca3cf3 fix document going out of scope 2019-11-19 14:41:08 +01:00
Jan Lukas Gernert
2c6bfed550 frickel 2019-11-18 05:53:34 +01:00
Jan Lukas Gernert
4b8af0d709 wip: async 2019-11-10 14:43:59 +01:00
Jan Lukas Gernert
5f82872d1f don't attempt to redownload embeded images 2019-09-26 21:48:24 +02:00
Jan Lukas Gernert
4f5aef8e17 merge 2019-09-26 21:29:11 +02:00
Jan Lukas Gernert
2137e84743 update to new serialization api of libxml 2019-09-26 21:28:05 +02:00
Jan Lukas Gernert
a99b8dec47 wip: test libxml XML_SAVE_NO_EMPTY option 2019-09-24 18:45:06 +02:00
Jan Lukas Gernert
a44ac3663c don't resize animated images 2019-09-24 14:25:57 +02:00
Jan Lukas Gernert
b489af74bd create data dir if it doesn't exist 2019-09-24 03:16:37 +02:00
Jan Lukas Gernert
481a2f41ac don't abort image download on failed image 2019-09-24 02:56:45 +02:00
Jan Lukas Gernert
f9905c8a9d download images parameter to parse method 2019-09-24 02:43:36 +02:00
Jan Lukas Gernert
f1be8a2608 make image downloader public 2019-09-24 02:40:43 +02:00
Jan Lukas Gernert
efd36dff66 update deps 2019-09-24 02:11:16 +02:00
Jan Lukas Gernert
4d2f5a0a50 update crates 2019-08-21 23:18:28 +02:00
Jan Lukas Gernert
fb6158f7df Merge branch 'embeded_images' into 'master' 
Embed images as base64 inside article html

See merge request news-flash/article_scraper!1
 2019-03-06 17:56:31 +00:00
Jan Lukas Gernert
3ca59d7f02 embed images as base64 inside article html 2019-03-06 18:37:24 +01:00
Jan Lukas Gernert
e1905d3c2c remove life time annotations added by rust 2018 2018-12-08 23:25:07 +01:00
Jan Lukas Gernert
02356a51aa fix save_html returning error even it succeeded 2018-12-07 15:31:09 +01:00
Jan Lukas Gernert
aa26e099df get rid of 'extern crate' 2018-12-07 02:26:46 +01:00
Jan Lukas Gernert
b679f2e1fa update to rust 2018 2018-12-07 02:19:40 +01:00
Jan Lukas Gernert
6f38c2bc4c merge 2018-12-07 02:17:15 +01:00
Jan Lukas Gernert
5555118914 update to reqwest 0.9 2018-12-07 02:15:06 +01:00
Jan Lukas Gernert
fcea6cf5d1 update to reqwest 0.9 2018-12-07 02:14:50 +01:00
Jan Lukas Gernert
fab4306ed9 TIL: map_err 2018-08-31 16:49:58 +02:00
Jan Lukas Gernert
5beb25a575 remove old dependency 2018-08-29 18:25:30 +02:00
Jan Lukas Gernert
b76bb7eea7 dont use .X versions of deps 2018-08-02 22:30:09 +02:00
Jan Lukas Gernert
ed66a705c1 update deps and remove html2text 2018-08-02 17:05:26 +02:00
Jan Lukas Gernert
4b2e6a24eb initial commit 2018-07-31 16:10:09 +02:00