Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
W
webmagic
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
沈俊林
webmagic
Commits
5a226387
Commit
5a226387
authored
Oct 11, 2013
by
yihua.huang
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
#27 nullpointer fix
parent
16e12e3b
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
8 additions
and
3 deletions
+8
-3
HttpClientPool.java
...java/us/codecraft/webmagic/downloader/HttpClientPool.java
+7
-3
HttpClientDownloaderTest.java
...decraft/webmagic/downloader/HttpClientDownloaderTest.java
+1
-0
No files found.
webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientPool.java
View file @
5a226387
...
...
@@ -51,9 +51,14 @@ public class HttpClientPool {
HttpParams
params
=
new
BasicHttpParams
();
if
(
site
!=
null
&&
site
.
getUserAgent
()
!=
null
)
{
params
.
setParameter
(
CoreProtocolPNames
.
USER_AGENT
,
site
.
getUserAgent
());
params
.
setIntParameter
(
CoreConnectionPNames
.
SO_TIMEOUT
,
site
.
getTimeOut
());
params
.
setIntParameter
(
CoreConnectionPNames
.
CONNECTION_TIMEOUT
,
site
.
getTimeOut
());
}
else
{
params
.
setIntParameter
(
CoreConnectionPNames
.
SO_TIMEOUT
,
3000
);
params
.
setIntParameter
(
CoreConnectionPNames
.
CONNECTION_TIMEOUT
,
3000
);
}
params
.
setIntParameter
(
CoreConnectionPNames
.
SO_TIMEOUT
,
site
.
getTimeOut
());
params
.
setIntParameter
(
CoreConnectionPNames
.
CONNECTION_TIMEOUT
,
site
.
getTimeOut
());
params
.
setParameter
(
ClientPNames
.
COOKIE_POLICY
,
CookiePolicy
.
BEST_MATCH
);
HttpProtocolParamBean
paramsBean
=
new
HttpProtocolParamBean
(
params
);
paramsBean
.
setVersion
(
HttpVersion
.
HTTP_1_1
);
...
...
@@ -73,7 +78,6 @@ public class HttpClientPool {
if
(
site
!=
null
)
{
generateCookie
(
httpClient
,
site
);
}
return
httpClient
;
}
...
...
webmagic-core/src/test/java/us/codecraft/webmagic/downloader/HttpClientDownloaderTest.java
View file @
5a226387
...
...
@@ -22,4 +22,5 @@ public class HttpClientDownloaderTest {
Page
download
=
httpClientDownloader
.
download
(
new
Request
(
"http://www.diandian.com"
),
site
.
toTask
());
Assert
.
assertTrue
(
download
.
getHtml
().
toString
().
contains
(
"flashsword30"
));
}
}
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment