Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
W
webmagic
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
沈俊林
webmagic
Commits
583a0eba
Commit
583a0eba
authored
Nov 03, 2013
by
yihua.huang
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
#29 refactor some method name
parent
6fa82a41
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
3 additions
and
3 deletions
+3
-3
Spider.java
...agic-core/src/main/java/us/codecraft/webmagic/Spider.java
+3
-3
No files found.
webmagic-core/src/main/java/us/codecraft/webmagic/Spider.java
View file @
583a0eba
...
@@ -343,12 +343,12 @@ public class Spider implements Runnable, Task {
...
@@ -343,12 +343,12 @@ public class Spider implements Runnable, Task {
}
}
// for cycle retry
// for cycle retry
if
(
page
.
getHtml
()
==
null
)
{
if
(
page
.
getHtml
()
==
null
)
{
addRequest
(
page
);
extractAndAddRequests
(
page
);
sleep
(
site
.
getSleepTime
());
sleep
(
site
.
getSleepTime
());
return
;
return
;
}
}
pageProcessor
.
process
(
page
);
pageProcessor
.
process
(
page
);
addRequest
(
page
);
extractAndAddRequests
(
page
);
if
(!
page
.
getResultItems
().
isSkip
())
{
if
(!
page
.
getResultItems
().
isSkip
())
{
for
(
Pipeline
pipeline
:
pipelines
)
{
for
(
Pipeline
pipeline
:
pipelines
)
{
pipeline
.
process
(
page
.
getResultItems
(),
this
);
pipeline
.
process
(
page
.
getResultItems
(),
this
);
...
@@ -365,7 +365,7 @@ public class Spider implements Runnable, Task {
...
@@ -365,7 +365,7 @@ public class Spider implements Runnable, Task {
}
}
}
}
protected
void
addRequest
(
Page
page
)
{
protected
void
extractAndAddRequests
(
Page
page
)
{
if
(
CollectionUtils
.
isNotEmpty
(
page
.
getTargetRequests
()))
{
if
(
CollectionUtils
.
isNotEmpty
(
page
.
getTargetRequests
()))
{
for
(
Request
request
:
page
.
getTargetRequests
())
{
for
(
Request
request
:
page
.
getTargetRequests
())
{
addRequest
(
request
);
addRequest
(
request
);
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment