Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
W
webmagic
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
沈俊林
webmagic
Commits
01f49aad
Commit
01f49aad
authored
Jun 16, 2013
by
yihua.huang
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
fix a pom error
parent
f57ada81
Changes
3
Show whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
16 additions
and
5 deletions
+16
-5
README.md
README.md
+11
-4
pom.xml
pom.xml
+1
-1
SimplePageProcessor.java
.../us/codecraft/webmagic/processor/SimplePageProcessor.java
+4
-0
No files found.
README.md
View file @
01f49aad
...
@@ -29,11 +29,9 @@ webmagic的功能覆盖整个爬虫的生命周期(链接提取、页面下载
...
@@ -29,11 +29,9 @@ webmagic的功能覆盖整个爬虫的生命周期(链接提取、页面下载
###Get Started
###Get Started
webmagic定制的核心是PageProcessor接口。
一个最简单的webmagic爬虫例子是这样的:
webmagic定制的核心是PageProcessor接口。
Spider.me().processor(new SimplePageProcessor("http://my.oschina.net/", "http://my.oschina.net/*/blog/*")).run();
例如,我们要实现一个简单的通用爬虫SimplePageProcessor,代码如下:
其中SimplePageProcessor实现如下:
public class SimplePageProcessor implements PageProcessor {
public class SimplePageProcessor implements PageProcessor {
...
@@ -53,17 +51,26 @@ webmagic定制的核心是PageProcessor接口。一个最简单的webmagic爬虫
...
@@ -53,17 +51,26 @@ webmagic定制的核心是PageProcessor接口。一个最简单的webmagic爬虫
@Override
@Override
public void process(Page page) {
public void process(Page page) {
List<String> requests = page.getHtml().as().rs(urlPattern).toStrings();
List<String> requests = page.getHtml().as().rs(urlPattern).toStrings();
//调用page.addTargetRequests()方法添加待抓取链接
page.addTargetRequests(requests);
page.addTargetRequests(requests);
//xpath方式抽取
page.putField("title", page.getHtml().x("//title"));
page.putField("title", page.getHtml().x("//title"));
//sc表示使用Readability技术抽取正文
page.putField("content", page.getHtml().sc());
page.putField("content", page.getHtml().sc());
}
}
@Override
@Override
public Site getSite() {
public Site getSite() {
//定义抽取站点的相关参数
return site;
return site;
}
}
}
}
调用这个爬虫的代码如下:
Spider.me().processor(new SimplePageProcessor("http://my.oschina.net/", "http://my.oschina.net/*/blog/*")).run();
### 示例
### 示例
可参考作者博客
[
使用webmagic抓取页面并保存为wordpress文件
](
http://my.oschina.net/flashsword/blog/136846
)
可参考作者博客
[
使用webmagic抓取页面并保存为wordpress文件
](
http://my.oschina.net/flashsword/blog/136846
)
...
...
pom.xml
View file @
01f49aad
...
@@ -5,7 +5,7 @@
...
@@ -5,7 +5,7 @@
<groupId>
us.codecraft
</groupId>
<groupId>
us.codecraft
</groupId>
<version>
0.0.1-SNAPSHOT
</version>
<version>
0.0.1-SNAPSHOT
</version>
<modelVersion>
4.0.0
</modelVersion>
<modelVersion>
4.0.0
</modelVersion>
<packaging>
pom
</packaging>
<artifactId>
webmagic
</artifactId>
<artifactId>
webmagic
</artifactId>
<modules>
<modules>
...
...
webmagic-core/src/main/java/us/codecraft/webmagic/processor/SimplePageProcessor.java
View file @
01f49aad
...
@@ -29,13 +29,17 @@ public class SimplePageProcessor implements PageProcessor {
...
@@ -29,13 +29,17 @@ public class SimplePageProcessor implements PageProcessor {
@Override
@Override
public
void
process
(
Page
page
)
{
public
void
process
(
Page
page
)
{
List
<
String
>
requests
=
page
.
getHtml
().
as
().
rs
(
urlPattern
).
toStrings
();
List
<
String
>
requests
=
page
.
getHtml
().
as
().
rs
(
urlPattern
).
toStrings
();
//调用page.addTargetRequests()方法添加待抓取链接
page
.
addTargetRequests
(
requests
);
page
.
addTargetRequests
(
requests
);
//xpath方式抽取
page
.
putField
(
"title"
,
page
.
getHtml
().
x
(
"//title"
));
page
.
putField
(
"title"
,
page
.
getHtml
().
x
(
"//title"
));
//sc表示使用Readability技术抽取正文
page
.
putField
(
"content"
,
page
.
getHtml
().
sc
());
page
.
putField
(
"content"
,
page
.
getHtml
().
sc
());
}
}
@Override
@Override
public
Site
getSite
()
{
public
Site
getSite
()
{
//定义抽取站点的相关参数
return
site
;
return
site
;
}
}
}
}
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment