Skip to content

W
webmagic
Commit ca072c55 authored by yihua.huang's avatar yihua.huang
Browse files
Options

fix URL regex in GithubRepoPageProcessor #305

parent 047cb8ff
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 2 deletions
webmagic-core/src/main/java/us/codecraft/webmagic/processor/example/GithubRepoPageProcessor.java
View file @ ca072c55
...@@ -15,8 +15,8 @@ public class GithubRepoPageProcessor implements PageProcessor { ...@@ -15,8 +15,8 @@ public class GithubRepoPageProcessor implements PageProcessor {
@Override @Override
public void process(Page page) { public void process(Page page) {
page.addTargetRequests(page.getHtml().links().regex("(https://github\\.com/\\w+/\\w+)").all()); page.addTargetRequests(page.getHtml().links().regex("(https://github\\.com/[\\w\\-]+/[\\w\\-])").all());
page.addTargetRequests(page.getHtml().links().regex("(https://github\\.com/\\w+)").all()); page.addTargetRequests(page.getHtml().links().regex("(https://github\\.com/[\\w\\-])").all());
page.putField("author", page.getUrl().regex("https://github\\.com/(\\w+)/.*").toString()); page.putField("author", page.getUrl().regex("https://github\\.com/(\\w+)/.*").toString());
page.putField("name", page.getHtml().xpath("//h1[@class='entry-title public']/strong/a/text()").toString()); page.putField("name", page.getHtml().xpath("//h1[@class='entry-title public']/strong/a/text()").toString());
if (page.getResultItems().get("name")==null){ if (page.getResultItems().get("name")==null){
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment