Skip to content

W
webmagic
Commit 6d78d51f authored by Jsbd's avatar Jsbd Committed by GitHub
Browse files
Options

Merge branch 'master' into master

parents 1b886d48 e3131af8
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 148 additions and 66639 deletions
README-zh.md
View file @ 6d78d51f
......@@ -6,9 +6,8 @@
官方网站[http://webmagic.io/](http://webmagic.io/)
>webmagic是一个开源的Java垂直爬虫框架，目标是简化爬虫的开发流程，让开发者专注于逻辑功能的开发。webmagic的核心非常简单，但是覆盖爬虫的整个流程，也是很好的学习爬虫开发的材料。作者曾经在前公司进行过一年的垂直爬虫的开发，webmagic就是为了解决爬虫开发的一些重复劳动而产生的框架。
>webmagic是一个开源的Java垂直爬虫框架，目标是简化爬虫的开发流程，让开发者专注于逻辑功能的开发。webmagic的核心非常简单，但是覆盖爬虫的整个流程，也是很好的学习爬虫开发的材料。
>web爬虫是一种技术，webmagic致力于将这种技术的实现成本降低，但是出于对资源提供者的尊重，webmagic不会做反封锁的事情，包括：验证码破解、代理切换、自动登录等。
webmagic的主要特色：
......@@ -39,12 +38,12 @@ webmagic使用maven管理依赖，在项目中添加对应的依赖即可使用w
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-core</artifactId>
<version>0.5.3</version>
<version>0.6.0</version>
</dependency>
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-extension</artifactId>
<version>0.5.3</version>
<version>0.6.0</version>
</dependency>
```
......
README.md
View file @ 6d78d51f
......@@ -2,8 +2,6 @@
[Readme in Chinese](https://github.com/code4craft/webmagic/tree/master/README-zh.md)
[User Manual (Chinese)](https://github.com/code4craft/webmagic/blob/master/user-manual.md)
[![Build Status](https://travis-ci.org/code4craft/webmagic.png?branch=master)](https://travis-ci.org/code4craft/webmagic)
......@@ -25,12 +23,12 @@ Add dependencies to your pom.xml:
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-core</artifactId>
<version>0.5.3</version>
<version>0.6.0</version>
</dependency>
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-extension</artifactId>
<version>0.5.3</version>
<version>0.6.0</version>
</dependency>
```
......
make.sh deleted 100644 → 0
View file @ 1b886d48
#!/bin/sh
mvn clean dependency:copy-dependencies -DoutputDirectory=target/lib
rsync -avz --delete ./webmagic-samples/target/lib/ ./lib/
pom.xml
View file @ 6d78d51f
......@@ -6,7 +6,7 @@
<version>7</version>
</parent>
<groupId>us.codecraft</groupId>
<version>0.5.4-SNAPSHOT</version>
<version>0.6.1-SNAPSHOT</version>
<modelVersion>4.0.0</modelVersion>
<packaging>pom</packaging>
<properties>
......@@ -38,7 +38,7 @@
<connection>scm:git:git@github.com:code4craft/webmagic.git</connection>
<developerConnection>scm:git:git@github.com:code4craft/webmagic.git</developerConnection>
<url>git@github.com:code4craft/webmagic.git</url>
<tag>webmagic-parent-0.5.4-SNAPSHOT</tag>
<tag>webmagic-parent-0.6.0</tag>
</scm>
<licenses>
<license>
......@@ -54,7 +54,6 @@
<module>webmagic-selenium</module>
<module>webmagic-saxon</module>
<module>webmagic-samples</module>
<module>webmagic-avalon</module>
</modules>
<dependencyManagement>
......@@ -68,13 +67,18 @@
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3.3</version>
<version>4.5.2</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>15.0</version>
</dependency>
<dependency>
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
<version>0.8.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
......
release.properties deleted 100644 → 0
View file @ 1b886d48
#release configuration
#Thu Jan 21 19:19:07 CST 2016
scm.commentPrefix=[maven-release-plugin]
pushChanges=true
project.dev.us.codecraft\:webmagic-extension=0.5.4-SNAPSHOT
project.scm.us.codecraft\:webmagic-parent.developerConnection=scm\:git\:git@github.com\:code4craft/webmagic.git
project.rel.us.codecraft\:webmagic-core=0.5.3
project.rel.us.codecraft\:webmagic-extension=0.5.3
project.rel.us.codecraft\:webmagic-parent=0.5.3
scm.tag=webmagic-parent-0.5.3
remoteTagging=true
project.scm.us.codecraft\:webmagic-parent.tag=webmagic-parent-0.5.3
exec.additionalArguments=-Psonatype-oss-release -P development
project.dev.us.codecraft\:webmagic-core=0.5.4-SNAPSHOT
scm.url=scm\:git\:git@github.com\:code4craft/webmagic.git
scm.tagNameFormat=@{project.artifactId}-@{project.version}
project.scm.us.codecraft\:webmagic-extension.empty=true
project.scm.us.codecraft\:webmagic-parent.url=git@github.com\:code4craft/webmagic.git
preparationGoals=clean verify
project.scm.us.codecraft\:webmagic-core.empty=true
project.scm.us.codecraft\:webmagic-parent.connection=scm\:git\:git@github.com\:code4craft/webmagic.git
exec.snapshotReleasePluginAllowed=false
project.dev.us.codecraft\:webmagic-parent=0.5.4-SNAPSHOT
completedPhase=end-release
user-manual.md deleted 100644 → 0
View file @ 1b886d48
This diff is collapsed.
webmagic-avalon.md deleted 100644 → 0
View file @ 1b886d48
WebMagic-Avalon项目计划
=======
WebMagic-Avalon项目的目标是打造一个可配置、可管理的爬虫，以及一个可分享配置/脚本的平台，从而减少熟悉的开发者的开发量，并且让**不熟悉Java技术的人**也能简单的使用一个爬虫。
## Part1:webmagic-scripts
目标：使得可以用简单脚本的方式编写爬虫，从而为一些常用场景提供可流通的脚本。
例如：我需要抓github的仓库数据，可以这样写一个脚本(javascript)：
[https://github.com/code4craft/webmagic/tree/master/webmagic-scripts](https://github.com/code4craft/webmagic/tree/master/webmagic-scripts)
这个功能目前实现了一部分，但最终结果仍在实验阶段。欢迎大家积极参与并提出意见。
## Part2:webmagic-pannel
一个集成了加载脚本、管理爬虫的后台。计划中。
## Part3:webmagic-market
一个可以分享、搜索和下载脚本的站点。计划中。
## 如何参与
webmagic目前由作者业余维护，仅仅为了分享和个人提高，没有任何盈利，也没有商业化打算。
欢迎以下几种形式的贡献：
1. 为webmagic项目本身提出改进意见，可以通过邮件组、qq、oschina或者在github提交issue(推荐)的方式。
2. 参与WebMagic-Avalon计划的建设讨论，包括产品设计、技术选型等，可以直接回复这个issue。
3. 参与webmagic代码开发，请fork一份代码，修改后提交pull request给我。请使用尽量新的版本，并说明修改内容。pull request接受后，我会将你加为committer，共同参与开发。
\ No newline at end of file
webmagic-avalon/README.md deleted 100644 → 0
View file @ 1b886d48
WebMagic-Avalon
========
> Spiders Manage Web
see [#issue43](https://github.com/code4craft/webmagic/issues/43)
\ No newline at end of file
webmagic-avalon/forger/LICENSE deleted 100644 → 0
View file @ 1b886d48
This diff is collapsed.
webmagic-avalon/forger/README.md deleted 100644 → 0
View file @ 1b886d48
forger
======
Dynamic Java object generator with template class and configuration.
## Compiler
Use groovy compiler. Compile source code to Java class.
## PropertyLoader
Load properties of object from user input.
## API
```java
@Test
public void testForgerCreateByClassAnnotationCompile() throws Exception {
ForgerFactory forgerFactory = new ForgerFactory(new AnnotationPropertyLoader(), new GroovyForgerCompiler());
Forger<Fooable> forger = forgerFactory.<Fooable>compile(Foo.SOURCE_CODE);
Fooable foo = forger.forge(ImmutableMap.<String, Object>of("fooa", "test"));
Field field = forger.getClazz().getDeclaredField("foo");
field.setAccessible(true);
assertThat(field.get(foo)).isEqualTo("test");
assertThat(foo.foo()).isEqualTo("test");
}
```
\ No newline at end of file
webmagic-avalon/forger/pom.xml deleted 100644 → 0
View file @ 1b886d48
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<groupId>org.sonatype.oss</groupId>
<artifactId>oss-parent</artifactId>
<version>7</version>
</parent>
<groupId>us.codecraft</groupId>
<artifactId>forger</artifactId>
<version>0.1.1-SNAPSHOT</version>
<modelVersion>4.0.0</modelVersion>
<packaging>jar</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>
<name>forger</name>
<description>
Dynamic Java object generator with template class and configuration.
</description>
<url>https://github.com/code4craft/forger/</url>
<developers>
<developer>
<id>code4craft</id>
<name>Yihua huang</name>
<email>code4crafer@gmail.com</email>
</developer>
</developers>
<scm>
<connection>scm:git:git@github.com:code4craft/forger.git</connection>
<developerConnection>scm:git:git@github.com:code4craft/forger.git</developerConnection>
<url>git@github.com:code4craft/forger.git</url>
<tag>HEAD</tag>
</scm>
<licenses>
<license>
<name>Apache License,Version 2</name>
<url>http://www.apache.org/licenses/LICENSE-2.0</url>
<distribution>repo</distribution>
</license>
</licenses>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.6</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.6</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>15.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.8</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<overWriteReleases>false</overWriteReleases>
<overWriteSnapshots>false</overWriteSnapshots>
<overWriteIfNewer>true</overWriteIfNewer>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<version>2.6</version>
<configuration>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
<configuration>
<encoding>UTF-8</encoding>
</configuration>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>2.4.1</version>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>release-sign-artifacts</id>
<activation>
<property>
<name>performRelease</name>
<value>true</value>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-gpg-plugin</artifactId>
<version>1.1</version>
<executions>
<execution>
<id>sign-artifacts</id>
<phase>verify</phase>
<goals>
<goal>sign</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
webmagic-avalon/forger/src/main/java/us/codecraft/forger/Forger.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger;
import us.codecraft.forger.property.Property;
import us.codecraft.forger.property.PropertyLoader;
import java.util.List;
import java.util.Map;
/**
* @author code4crafter@gmail.com
*/
public class Forger<T> {
private final Class<T> clazz;
private final PropertyLoader propertyLoader;
public Forger(Class<T> clazz,PropertyLoader propertyLoader) {
this.clazz = clazz;
this.propertyLoader = propertyLoader;
}
public T forge(Map<String, Object> properties) throws IllegalAccessException, InstantiationException {
T t = clazz.newInstance();
propertyLoader.load(t, properties);
return t;
}
public List<Property> getPropertyNames() {
return propertyLoader.getProperties(clazz);
}
public Class<T> getClazz() {
return clazz;
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/ForgerFactory.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger;
import us.codecraft.forger.compiler.ForgerCompiler;
import us.codecraft.forger.property.PropertyLoader;
/**
* @author code4crafter@gmail.com
*/
public class ForgerFactory {
private final PropertyLoader propertyLoader;
private final ForgerCompiler forgerCompiler;
public ForgerFactory(PropertyLoader propertyLoader, ForgerCompiler forgerCompiler) {
this.propertyLoader = propertyLoader;
this.forgerCompiler = forgerCompiler;
}
public <T> Forger<T> compile(String sourceCode) {
Class clazz = forgerCompiler.compile(sourceCode);
return new Forger(clazz, propertyLoader);
}
public <T> Forger<T> create(Class clazz) {
return new Forger(clazz, propertyLoader);
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/compiler/ForgerCompiler.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.compiler;
/**
* @author code4crafter@gmail.com
*/
public interface ForgerCompiler {
public Class compile(String sourceCode);
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/compiler/GroovyForgerCompiler.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.compiler;
import groovy.lang.GroovyClassLoader;
/**
* @author code4crafter@gmail.com
*/
public class GroovyForgerCompiler implements ForgerCompiler{
private GroovyClassLoader groovyClassLoader = new GroovyClassLoader();
@Override
public Class compile(String sourceCode) {
return groovyClassLoader.parseClass(sourceCode);
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/AbstractPropertyLoader.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import us.codecraft.forger.property.format.*;
import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
* @author code4crafter@gmail.com
*/
public abstract class AbstractPropertyLoader implements PropertyLoader {
private TypeFormatterFactory typeFormatterFactory = new TypeFormatterFactory();
protected Logger logger = LoggerFactory.getLogger(getClass());
protected TypeFormatterFactory getTypeFormatterFactory() {
return typeFormatterFactory;
}
@Override
public <T> T load(T object, Map<String, Object> propertyConfigs) {
List<Property> properties = getProperties(object.getClass());
for (Property property : properties) {
Object value = propertyConfigs.get(property.getName());
if (value == null) {
throw new IllegalArgumentException("Config for property " + property.getName() + " is missing!");
}
ObjectFormatter objectFormatter = property.getObjectFormatter();
switch (property.getType()) {
case PropertyString:
Object fieldValue = objectFormatter.format(String.valueOf(value));
try {
property.getField().set(object, fieldValue);
} catch (IllegalAccessException e) {
logger.warn("Set field " + property.getField() + " error!", e);
}
break;
case PropertyList:
if (!List.class.isAssignableFrom(value.getClass())) {
throw new IllegalArgumentException("Config for property " + property.getName() + " should be subclass of List!");
}
List listField = new ArrayList();
List<String> listConfigs = (List) value;
for (String listConfig : listConfigs) {
listField.add(objectFormatter.format(listConfig));
}
try {
property.getField().set(object, listField);
} catch (IllegalAccessException e) {
logger.warn("Set field " + property.getField() + " error!", e);
}
break;
case PropertyMap:
if (!Map.class.isAssignableFrom(value.getClass())) {
throw new IllegalArgumentException("Config for property " + property.getName() + " should be subclass of List!");
}
Map mapField = new HashMap();
Map<String, String> mapConfigs = (Map<String, String>) value;
for (Map.Entry<String, String> entry : mapConfigs.entrySet()) {
mapField.put(entry.getKey(), objectFormatter.format(entry.getValue()));
}
try {
property.getField().set(object, mapField);
} catch (IllegalAccessException e) {
logger.warn("Set field " + property.getField() + " error!", e);
}
break;
}
}
return object;
}
protected ObjectFormatter prepareTypeFormatterParam(TypeFormatter objectFormatter, String[] params) {
if (params == null) {
return objectFormatter;
}
return new ObjectFormatterWithParams().setTypeFormatter(objectFormatter).setParams(params);
}
protected ObjectFormatter getObjectFormatter(Field field) {
Class type = field.getType();
if (List.class.isAssignableFrom(type) || Map.class.isAssignableFrom(type)) {
type = String.class;
}
if (field.isAnnotationPresent(Formatter.class)) {
Formatter formatter = field.getAnnotation(Formatter.class);
if (!formatter.formatter().equals(TypeFormatter.class)) {
TypeFormatter typeFormatter = typeFormatterFactory.getByFormatterClass(formatter.formatter());
if (typeFormatter != null) {
return prepareTypeFormatterParam(typeFormatter,formatter.value());
}
typeFormatterFactory.put(formatter.formatter());
return prepareTypeFormatterParam(typeFormatterFactory.getByFormatterClass(formatter.formatter()), formatter.value());
} else if (!formatter.subClazz().equals(String.class)) {
type = formatter.subClazz();
TypeFormatter typeFormatter = typeFormatterFactory.get(type);
if (typeFormatter == null) {
throw new IllegalArgumentException("No typeFormatter for class " + type);
}
return prepareTypeFormatterParam(typeFormatter, formatter.value());
}
}
return getTypeFormatterFactory().get(BasicTypeFormatter.detectBasicClass(type));
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/AnnotationPropertyLoader.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property;
import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.List;
/**
* @author code4crafter@gmail.com
*/
public class AnnotationPropertyLoader extends AbstractPropertyLoader {
@Override
public List<Property> getProperties(Class clazz) {
Field[] fields = clazz.getDeclaredFields();
List<Property> properties = new ArrayList<Property>(fields.length);
for (Field field : fields) {
Inject inject = field.getAnnotation(Inject.class);
if (inject != null) {
if (!field.isAccessible()) {
field.setAccessible(true);
}
Property property = Property.fromField(field);
if (inject.value().length() > 0) {
property.setName(inject.value());
}
property.setObjectFormatter(getObjectFormatter(field));
properties.add(property);
}
}
return properties;
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/Inject.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.Target;
/**
* @author code4crafter@gmail.com
*/
@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({ElementType.FIELD})
public @interface Inject {
String value() default "";
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/Property.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property;
import us.codecraft.forger.property.format.ObjectFormatter;
import java.lang.reflect.Field;
/**
* @author code4crafter@gmail.com
*/
public class Property {
private String name;
private PropertyType type;
private Field field;
private ObjectFormatter objectFormatter;
public ObjectFormatter getObjectFormatter() {
return objectFormatter;
}
public Property setObjectFormatter(ObjectFormatter objectFormatter) {
this.objectFormatter = objectFormatter;
return this;
}
public String getName() {
return name;
}
public Property setName(String name) {
this.name = name;
return this;
}
public PropertyType getType() {
return type;
}
public Property setType(PropertyType type) {
this.type = type;
return this;
}
public Field getField() {
return field;
}
public Property setField(Field field) {
this.field = field;
return this;
}
public static Property fromField(Field field) {
return new Property().setName(field.getName()).setType(PropertyType.from(field.getType())).setField(field);
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/PropertyLoader.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property;
import java.util.List;
import java.util.Map;
/**
* @author code4crafter@gmail.com
*/
public interface PropertyLoader {
public <T> T load(T object, Map<String, Object> propertyConfigs);
public List<Property> getProperties(Class clazz);
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/PropertyType.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property;
import java.util.List;
import java.util.Map;
/**
* @author code4crafter@gmail.com
*/
public enum PropertyType {
PropertyString,PropertyMap,PropertyList;
public static PropertyType from(Class clazz){
if (Map.class.isAssignableFrom(clazz)){
return PropertyMap;
}
if (List.class.isAssignableFrom(clazz)){
return PropertyList;
}
return PropertyString;
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/SimpleFieldPropertyLoader.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property;
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
import java.util.ArrayList;
import java.util.List;
/**
* @author code4crafter@gmail.com
*/
public class SimpleFieldPropertyLoader extends AbstractPropertyLoader {
@Override
public List<Property> getProperties(Class clazz) {
Field[] fields = clazz.getDeclaredFields();
List<Property> properties = new ArrayList<Property>(fields.length);
for (Field field : fields) {
if (Modifier.isStatic(field.getModifiers())){
continue;
}
if (!field.isAccessible()){
field.setAccessible(true);
}
properties.add(Property.fromField(field).setObjectFormatter(getObjectFormatter(field)));
}
return properties;
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/BasicTypeFormatter.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property.format;
import java.util.Arrays;
import java.util.List;
/**
* @author code4crafter@gmail.com
* @since 0.3.2
*/
public abstract class BasicTypeFormatter<T> implements TypeFormatter<T> {
@Override
public T format(String text) {
if (text == null) {
return null;
}
text = text.trim();
return formatTrimmed(text);
}
@Override
public T format(String text, String[] params) {
return format(text);
}
protected abstract T formatTrimmed(String raw);
public static final List<Class<? extends TypeFormatter>> basicTypeFormatters = Arrays.<Class<? extends TypeFormatter>>asList(IntegerFormatter.class,
LongFormatter.class, DoubleFormatter.class, FloatFormatter.class, ShortFormatter.class,
CharactorFormatter.class, ByteFormatter.class, BooleanFormatter.class, DateFormatter.class, StringFormatter.class);
public static Class<?> detectBasicClass(Class<?> type) {
if (type.equals(Integer.TYPE) || type.equals(Integer.class)) {
return Integer.class;
} else if (type.equals(Long.TYPE) || type.equals(Long.class)) {
return Long.class;
} else if (type.equals(Double.TYPE) || type.equals(Double.class)) {
return Double.class;
} else if (type.equals(Float.TYPE) || type.equals(Float.class)) {
return Float.class;
} else if (type.equals(Short.TYPE) || type.equals(Short.class)) {
return Short.class;
} else if (type.equals(Character.TYPE) || type.equals(Character.class)) {
return Character.class;
} else if (type.equals(Byte.TYPE) || type.equals(Byte.class)) {
return Byte.class;
} else if (type.equals(Boolean.TYPE) || type.equals(Boolean.class)) {
return Boolean.class;
}
return type;
}
public static class IntegerFormatter extends BasicTypeFormatter<Integer> {
@Override
public Integer formatTrimmed(String raw) {
return Integer.parseInt(raw);
}
@Override
public Class<Integer> clazz() {
return Integer.class;
}
}
public static class LongFormatter extends BasicTypeFormatter<Long> {
@Override
public Long formatTrimmed(String raw) {
return Long.parseLong(raw);
}
@Override
public Class<Long> clazz() {
return Long.class;
}
}
public static class DoubleFormatter extends BasicTypeFormatter<Double> {
@Override
public Double formatTrimmed(String raw) {
return Double.parseDouble(raw);
}
@Override
public Class<Double> clazz() {
return Double.class;
}
}
public static class FloatFormatter extends BasicTypeFormatter<Float> {
@Override
public Float formatTrimmed(String raw) {
return Float.parseFloat(raw);
}
@Override
public Class<Float> clazz() {
return Float.class;
}
}
public static class ShortFormatter extends BasicTypeFormatter<Short> {
@Override
public Short formatTrimmed(String raw) {
return Short.parseShort(raw);
}
@Override
public Class<Short> clazz() {
return Short.class;
}
}
public static class CharactorFormatter extends BasicTypeFormatter<Character> {
@Override
public Character formatTrimmed(String raw) {
return raw.charAt(0);
}
@Override
public Class<Character> clazz() {
return Character.class;
}
}
public static class ByteFormatter extends BasicTypeFormatter<Byte> {
@Override
public Byte formatTrimmed(String raw) {
return Byte.parseByte(raw, 10);
}
@Override
public Class<Byte> clazz() {
return Byte.class;
}
}
public static class BooleanFormatter extends BasicTypeFormatter<Boolean> {
@Override
public Boolean formatTrimmed(String raw) {
return Boolean.parseBoolean(raw);
}
@Override
public Class<Boolean> clazz() {
return Boolean.class;
}
}
public static class StringFormatter implements TypeFormatter<String> {
@Override
public String format(String text) {
return text;
}
@Override
public String format(String text, String[] params) {
return format(text);
}
@Override
public Class<String> clazz() {
return String.class;
}
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/DateFormatter.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property.format;
import org.apache.commons.lang3.time.DateUtils;
import java.text.ParseException;
import java.util.Date;
/**
* @author code4crafter@gmail.com
* @since 0.3.2
*/
public class DateFormatter implements TypeFormatter<Date> {
public static final String[] DEFAULT_PATTERN = new String[]{"yyyy-MM-dd HH:mm"};
@Override
public Date format(String text) {
return format(text,DEFAULT_PATTERN);
}
@Override
public Date format(String text, String[] params) {
try {
return DateUtils.parseDate(text, params);
} catch (ParseException e) {
throw new IllegalArgumentException(e);
}
}
@Override
public Class<Date> clazz() {
return Date.class;
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/Formatter.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property.format;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.Target;
/**
* Define how the result string is convert to an object for field.
*
* @author code4crafter@gmail.com <br>
* @since 0.3.2
*/
@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({ElementType.FIELD})
public @interface Formatter {
/**
* Set formatter params.
*
* @return formatter params
*/
String[] value();
/**
* Specific the class of field of class of elements in collection for field. <br>
* It is not necessary to be set because we can detect the class by class of field,
* unless you use a collection as a field. <br>
*
* @return the class of field
*/
Class subClazz() default String.class;
/**
* If there are more than one formatter for a class, just specify the implement.
* @return implement
*/
Class<? extends TypeFormatter> formatter() default TypeFormatter.class;
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/ObjectFormatter.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property.format;
/**
* @author code4crafter@gmail.com
*/
public interface ObjectFormatter<T> {
T format(String text);
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/ObjectFormatterWithParams.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property.format;
/**
* @author code4crafter@gmail.com
*/
public class ObjectFormatterWithParams<T> implements ObjectFormatter<T> {
private TypeFormatter<T> typeFormatter;
private String[] params;
public TypeFormatter getTypeFormatter() {
return typeFormatter;
}
public ObjectFormatterWithParams<T> setTypeFormatter(TypeFormatter typeFormatter) {
this.typeFormatter = typeFormatter;
return this;
}
public String[] getParams() {
return params;
}
public ObjectFormatterWithParams setParams(String[] params) {
this.params = params;
return this;
}
@Override
public T format(String text) {
return typeFormatter.format(text, params);
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/TypeFormatter.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property.format;
/**
* @author code4crafter@gmail.com
*/
public interface TypeFormatter<T> extends ObjectFormatter<T> {
T format(String text, String[] params);
Class<T> clazz();
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/TypeFormatterFactory.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.property.format;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
/**
* @author code4crafter@gmail.com
* @since 0.3.2
*/
public class TypeFormatterFactory {
private Logger logger = LoggerFactory.getLogger(getClass());
private Map<Class, TypeFormatter> objectFormatterMapWithPropertyAsKey = new ConcurrentHashMap<Class, TypeFormatter>();
private Map<Class, TypeFormatter> objectFormatterMapWithClassAsKey = new ConcurrentHashMap<Class, TypeFormatter>();
public TypeFormatterFactory() {
initFormatterMap();
}
private void initFormatterMap() {
for (Class<? extends TypeFormatter> basicTypeFormatter : BasicTypeFormatter.basicTypeFormatters) {
put(basicTypeFormatter);
}
put(DateFormatter.class);
}
public synchronized void put(Class<? extends TypeFormatter> objectFormatterClazz) {
try {
TypeFormatter typeFormatter = objectFormatterClazz.newInstance();
if (typeFormatter.clazz() != null) {
objectFormatterMapWithPropertyAsKey.put(typeFormatter.clazz(), typeFormatter);
}
objectFormatterMapWithClassAsKey.put(objectFormatterClazz, typeFormatter);
} catch (InstantiationException e) {
logger.error("Init objectFormatter error", e);
} catch (IllegalAccessException e) {
logger.error("Init objectFormatter error", e);
}
}
public TypeFormatter get(Class<?> clazz) {
return objectFormatterMapWithPropertyAsKey.get(clazz);
}
public TypeFormatter getByFormatterClass(Class<?> clazz) {
return objectFormatterMapWithClassAsKey.get(clazz);
}
}
webmagic-avalon/forger/src/main/resources/log4j.xml deleted 100644 → 0
View file @ 1b886d48
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yy-MM-dd HH:mm:ss,SSS} %-5p %c(%F:%L) ## %m%n" />
</layout>
</appender>
<logger name="org.apache" additivity="false">
<level value="warn" />
<appender-ref ref="stdout" />
</logger>
<root>
<level value="info" />
<appender-ref ref="stdout" />
</root>
</log4j:configuration>
webmagic-avalon/forger/src/test/java/us/codecraft/forger/Bar.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger;
import us.codecraft.forger.property.Inject;
import us.codecraft.forger.property.format.Formatter;
import java.util.List;
import java.util.Map;
/**
* @author code4crafter@gmail.com
*/
public class Bar {
@Inject("bar")
private String bar;
@Inject
private List<String> values;
@Formatter(value = "", subClazz = Integer.class)
@Inject
private Map<String, Integer> idMap;
public String getBar() {
return bar;
}
public void setBar(String bar) {
this.bar = bar;
}
public List<String> getValues() {
return values;
}
public void setValues(List<String> values) {
this.values = values;
}
public Map<String, Integer> getIdMap() {
return idMap;
}
public void setIdMap(Map<String, Integer> idMap) {
this.idMap = idMap;
}
}
webmagic-avalon/forger/src/test/java/us/codecraft/forger/Foo.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger;
import us.codecraft.forger.property.Inject;
import us.codecraft.forger.property.format.Formatter;
/**
* @author code4crafter@gmail.com
*/
public class Foo implements Fooable{
@Formatter("")
@Inject("fooa")
private String foo;
public static final String SOURCE_CODE="import us.codecraft.forger.*;\n" +
"import us.codecraft.forger.property.Inject;\n" +
"import us.codecraft.forger.property.Inject;\n" +
"import us.codecraft.forger.property.format.Formatter;\n" +
"\n" +
"/**\n" +
" * @author code4crafter@gmail.com\n" +
" */\n" +
"public class Foo implements Fooable{\n" +
"\n" +
" @Formatter(\"\")\n" +
" @Inject(\"fooa\")\n" +
" private String foo;\n" +
"\n" +
" public String getFoo() {\n" +
" return foo;\n" +
" }\n" +
"\n" +
" @Override\n" +
" public String foo() {\n" +
" return foo;\n" +
" }\n" +
"}";
public String getFoo() {
return foo;
}
@Override
public String foo() {
return foo;
}
}
webmagic-avalon/forger/src/test/java/us/codecraft/forger/Fooable.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger;
/**
* @author code4crafter@gmail.com
*/
public interface Fooable {
public String foo();
}
webmagic-avalon/forger/src/test/java/us/codecraft/forger/ForgerFactoryTest.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger;
import com.google.common.collect.ImmutableMap;
import org.junit.Test;
import us.codecraft.forger.compiler.GroovyForgerCompiler;
import us.codecraft.forger.property.AnnotationPropertyLoader;
import us.codecraft.forger.property.SimpleFieldPropertyLoader;
import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.*;
/**
* @author code4crafter@gmail.com
*/
public class ForgerFactoryTest {
@Test
public void testForgerCreateByClassProperty() throws Exception {
ForgerFactory forgerFactory = new ForgerFactory(new SimpleFieldPropertyLoader(), null);
Forger<Foo> forger = forgerFactory.<Foo>create(Foo.class);
Foo foo = forger.forge(ImmutableMap.<String, Object>of("foo", "test"));
assertThat(foo.getFoo()).isEqualTo("test");
}
@Test
public void testForgerCreateByClassAnnotation() throws Exception {
ForgerFactory forgerFactory = new ForgerFactory(new AnnotationPropertyLoader(), null);
Forger<Foo> forger = forgerFactory.<Foo>create(Foo.class);
Foo foo = forger.forge(ImmutableMap.<String, Object>of("fooa", "test"));
assertThat(foo.getFoo()).isEqualTo("test");
}
@Test
public void testForgerCreateByClassAnnotationCompile() throws Exception {
ForgerFactory forgerFactory = new ForgerFactory(new AnnotationPropertyLoader(), new GroovyForgerCompiler());
Forger<Fooable> forger = forgerFactory.<Fooable>compile(Foo.SOURCE_CODE);
Fooable foo = forger.forge(ImmutableMap.<String, Object>of("fooa", "test"));
Field field = forger.getClazz().getDeclaredField("foo");
field.setAccessible(true);
assertThat(field.get(foo)).isEqualTo("test");
assertThat(foo.foo()).isEqualTo("test");
}
@Test
public void testForgerCreateByClassAnnotationWithCollections() throws Exception {
ForgerFactory forgerFactory = new ForgerFactory(new AnnotationPropertyLoader(), null);
Forger<Bar> forger = forgerFactory.<Bar>create(Bar.class);
Map<String, Object> map = new HashMap<String, Object>();
map.put("bar", "bar");
Map<String, String> submap = new HashMap<String, String>();
submap.put("1", "1");
submap.put("2", "2");
map.put("idMap", submap);
List<String> sublist = new ArrayList<String>();
sublist.add("test");
map.put("values", sublist);
Bar forge = forger.forge(map);
assertThat(forge.getValues().size() > 0);
assertThat(forge.getIdMap().get("1")).isEqualTo(1);
}
}
webmagic-avalon/forger/src/test/java/us/codecraft/forger/compiler/GroovyForgerCompilerTest.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.forger.compiler;
import org.junit.Test;
import us.codecraft.forger.Foo;
import static org.assertj.core.api.Assertions.assertThat;
/**
* @author code4crafter@gmail.com
*/
public class GroovyForgerCompilerTest {
@Test
public void testGroovyClassLoader() throws Exception {
GroovyForgerCompiler groovyForgerCompiler = new GroovyForgerCompiler();
Class compiledClass = groovyForgerCompiler.compile(Foo.SOURCE_CODE);
assertThat(compiledClass.getName()).isEqualTo("Foo");
}
}
webmagic-avalon/forger/src/test/resources/log4j.xml deleted 100644 → 0
View file @ 1b886d48
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yy-MM-dd HH:mm:ss,SSS} %-5p %c(%F:%L) ## %m%n" />
</layout>
</appender>
<logger name="org.springframework" additivity="false">
<level value="warn" />
<appender-ref ref="stdout" />
</logger>
<logger name="org.apache" additivity="false">
<level value="warn" />
<appender-ref ref="stdout" />
</logger>
<logger name="net.sf.ehcache" additivity="false">
<level value="warn" />
<appender-ref ref="stdout" />
</logger>
<root>
<level value="info" />
<appender-ref ref="stdout" />
</root>
</log4j:configuration>
webmagic-avalon/pom.xml deleted 100644 → 0
View file @ 1b886d48
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>webmagic-parent</artifactId>
<groupId>us.codecraft</groupId>
<version>0.5.4-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-avalon</artifactId>
<packaging>pom</packaging>
<modules>
<module>forger</module>
<module>webmagic-admin</module>
<module>webmagic-worker</module>
<module>webmagic-avalon-common</module>
</modules>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-scripts</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
<version>3.1.1</version>
</dependency>
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis-spring</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>org.freemarker</groupId>
<artifactId>freemarker</artifactId>
<version>2.3.19</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>${spring-version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>1.5.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.18</version>
</dependency>
<dependency>
<groupId>commons-dbcp</groupId>
<artifactId>commons-dbcp</artifactId>
<version>1.3</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aop</artifactId>
<version>${spring-version}</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjrt</artifactId>
<version>1.7.2</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>1.7.2</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${spring-version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>${spring-version}</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>${spring-version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context-support</artifactId>
<version>${spring-version}</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.1.37</version>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>
</project>
webmagic-avalon/webmagic-admin/README.md deleted 100644 → 0
View file @ 1b886d48
WebMagic-Admin
=====
Admin is the control web of workers.
\ No newline at end of file
webmagic-avalon/webmagic-admin/pom.xml deleted 100644 → 0
View file @ 1b886d48
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>webmagic-avalon</artifactId>
<groupId>us.codecraft</groupId>
<version>0.5.4-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-admin</artifactId>
<packaging>war</packaging>
<dependencies>
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-avalon-common</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>
</project>
\ No newline at end of file
webmagic-avalon/webmagic-admin/src/main/java/us/codecraft/webmagic/avalon/web/DashBoardController.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.webmagic.avalon.web;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.servlet.ModelAndView;
/**
* @author code4crafter@gmail.com
*/
@Controller("dashboard")
@RequestMapping("/")
public class DashBoardController {
@RequestMapping
public ModelAndView index() {
ModelAndView map = new ModelAndView("dashboard");
return map;
}
}
webmagic-avalon/webmagic-admin/src/main/java/us/codecraft/webmagic/avalon/web/SpiderController.java deleted 100644 → 0
View file @ 1b886d48
package us.codecraft.webmagic.avalon.web;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import java.util.HashMap;
import java.util.Map;
/**
* @author code4crafter@gmail.com
*/
@Controller("spider")
@RequestMapping("spider")
public class SpiderController {
@RequestMapping("create")
@ResponseBody
public Map<String, Object> create() {
HashMap<String, Object> map = new HashMap<String, Object>();
map.put("code", 200);
return map;
}
}
webmagic-avalon/webmagic-admin/src/main/resources/freemarker.properties deleted 100644 → 0
View file @ 1b886d48
number_format=#
classic_compatible=true
default_encoding=UTF-8
template_update_delay=0
#########################
template_exception_handler=rethrow
webmagic-avalon/webmagic-admin/src/main/resources/log/log4j.xml deleted 100644 → 0
View file @ 1b886d48
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yy-MM-dd HH:mm:ss,SSS} %-5p %c(%F:%L) ## %m%n" />
</layout>
</appender>
<logger name="org.apache" additivity="false">
<level value="warn" />
<appender-ref ref="stdout" />
</logger>
<root>
<level value="info" />
<appender-ref ref="stdout" />
</root>
</log4j:configuration>
webmagic-avalon/webmagic-admin/src/main/webapp/WEB-INF/jsp/500.jsp deleted 100644 → 0
View file @ 1b886d48
<%@ page language="java" contentType="text/html; charset=utf8"
pageEncoding="utf8" isErrorPage="true" import="java.io.*"%>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf8">
<title>500</title>
</head>
<body>
页面出错啦！
<%
StringWriter stringWriter = new StringWriter();
exception.printStackTrace(new PrintWriter(stringWriter));
out.println(stringWriter.toString());
%>
</body>
</html>
\ No newline at end of file
webmagic-avalon/webmagic-admin/src/main/webapp/WEB-INF/pages/create_spider.ftl deleted 100644 → 0
View file @ 1b886d48
<html>
<head>
<script src=""></script>
</head>
<div class="url-box">
<input type="text" id="url-input">
</div>
<div class="content-show">
</div>
</html>
\ No newline at end of file
webmagic-avalon/webmagic-admin/src/main/webapp/WEB-INF/web.xml deleted 100644 → 0
View file @ 1b886d48
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">
<display-name>Archetype Created Web Application</display-name>
<context-param>
<param-name>contextConfigLocation</param-name>
<param-value>
classpath*:/config/spring/applicationContext*.xml,
</param-value>
</context-param>
<context-param>
<param-name>contextClass</param-name>
<param-value>org.springframework.web.context.support.XmlWebApplicationContext</param-value>
</context-param>
<!--由Sprng载入的Log4j配置文件位置 -->
<context-param>
<param-name>log4jConfigLocation</param-name>
<param-value>classpath:log/log4j.xml</param-value>
</context-param>
<context-param>
<param-name>log4jRefreshInterval</param-name>
<param-value>60000</param-value>
</context-param>
<servlet>
<servlet-name>spring</servlet-name>
<servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
<init-param>
<param-name>contextConfigLocation</param-name>
<param-value>classpath*:config/spring/applicationContext*.xml</param-value>
</init-param>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>spring</servlet-name>
<url-pattern>/</url-pattern>
</servlet-mapping>
<error-page>
<error-code>404</error-code>
<location>/WEB-INF/jsp/404.jsp</location>
</error-page>
<error-page>
<error-code>500</error-code>
<location>/WEB-INF/jsp/500.jsp</location>
</error-page>
</web-app>
webmagic-avalon/webmagic-admin/src/main/webapp/static/css/colorbox.css deleted 100755 → 0
View file @ 1b886d48
/*
ColorBox Core Style:
The following CSS is consistent between example themes and should not be altered.
*/
#colorbox, #cboxOverlay, #cboxWrapper{position:absolute; top:0; left:0; z-index:9999; overflow:hidden;}
#cboxOverlay{position:fixed; width:100%; height:100%;}
#cboxMiddleLeft, #cboxBottomLeft{clear:left;}
#cboxContent{position:relative;}
#cboxLoadedContent{overflow:auto;}
#cboxTitle{margin:0;}
#cboxLoadingOverlay, #cboxLoadingGraphic{position:absolute; top:0; left:0; width:100%; height:100%;}
#cboxPrevious, #cboxNext, #cboxClose, #cboxSlideshow{cursor:pointer;}
.cboxPhoto{float:left; margin:auto; border:0; display:block; max-width:none;}
.cboxIframe{width:100%; height:100%; display:block; border:0;}
#colorbox, #cboxContent, #cboxLoadedContent{box-sizing:content-box;}
/*
User Style:
Change the following styles to modify the appearance of ColorBox. They are
ordered & tabbed in a way that represents the nesting of the generated HTML.
*/
#cboxOverlay{background:#000;}
#colorbox{}
#cboxTopLeft{width:14px; height:14px; background:url(../img/controls.png) no-repeat 0 0;}
#cboxTopCenter{height:14px; background:url(../img/border.png) repeat-x top left;}
#cboxTopRight{width:14px; height:14px; background:url(../img/controls.png) no-repeat -36px 0;}
#cboxBottomLeft{width:14px; height:43px; background:url(../img/controls.png) no-repeat 0 -32px;}
#cboxBottomCenter{height:43px; background:url(../img/border.png) repeat-x bottom left;}
#cboxBottomRight{width:14px; height:43px; background:url(../img/controls.png) no-repeat -36px -32px;}
#cboxMiddleLeft{width:14px; background:url(../img/controls.png) repeat-y -175px 0;}
#cboxMiddleRight{width:14px; background:url(../img/controls.png) repeat-y -211px 0;}
#cboxContent{background:#fff; overflow:visible;}
.cboxIframe{background:#fff;}
#cboxError{padding:50px; border:1px solid #ccc;}
#cboxLoadedContent{margin-bottom:5px;}
#cboxLoadingOverlay{background:url(../img/loading_background.png) no-repeat center center;}
#cboxLoadingGraphic{background:url(../img/loading.gif) no-repeat center center;}
#cboxTitle{position:absolute; bottom:-25px; left:0; text-align:center; width:100%; font-weight:bold; color:#7C7C7C;}
#cboxCurrent{position:absolute; bottom:-25px; left:58px; font-weight:bold; color:#7C7C7C;}
#cboxPrevious, #cboxNext, #cboxClose, #cboxSlideshow{position:absolute; bottom:-29px; background:url(../img/controls.png) no-repeat 0px 0px; width:23px; height:23px; text-indent:-9999px;}
#cboxPrevious{left:0px; background-position: -51px -25px;}
#cboxPrevious:hover{background-position:-51px 0px;}
#cboxNext{left:27px; background-position:-75px -25px;}
#cboxNext:hover{background-position:-75px 0px;}
#cboxClose{right:0; background-position:-100px -25px;}
#cboxClose:hover{background-position:-100px 0px;}
.cboxSlideshow_on #cboxSlideshow{background-position:-125px 0px; right:27px;}
.cboxSlideshow_on #cboxSlideshow:hover{background-position:-150px 0px;}
.cboxSlideshow_off #cboxSlideshow{background-position:-150px -25px; right:27px;}
.cboxSlideshow_off #cboxSlideshow:hover{background-position:-125px 0px;}
webmagic-avalon/webmagic-admin/src/main/webapp/static/css/elfinder.theme.css deleted 100755 → 0
View file @ 1b886d48
/**
* MacOS X like theme for elFinder.
* Required jquery ui "smoothness" theme.
*
* @author Dmitry (dio) Levashov
**/
/* dialogs */
.std42-dialog, .std42-dialog .ui-widget-content { background-color:#ededed; background-image:none; background-clip: content-box; }
/* navbar */
.elfinder .elfinder-navbar { background:#dde4eb; }
.elfinder-navbar .ui-state-hover { background:transparent; border-color:transparent; }
.elfinder-navbar .ui-state-active { background: #3875d7; border-color:#3875d7; color:#fff; }
/* disabled elfinder */
.elfinder-disabled .elfinder-navbar .ui-state-active { background: #dadada; border-color:#aaa; color:#fff; }
/* current directory */
/* selected file in "icons" view */
.elfinder-cwd-view-icons .elfinder-cwd-file .ui-state-hover { background:#ccc; }
/* list view*/
.elfinder-cwd table tr:nth-child(odd) { background-color:#edf3fe; }
.elfinder-cwd table tr { border-top:1px solid #fff; }
/* common selected background/color */
.elfinder-cwd-view-icons .elfinder-cwd-file .elfinder-cwd-filename.ui-state-hover,
.elfinder-cwd table td.ui-state-hover,
.elfinder-button-menu .ui-state-hover { background: #3875d7; color:#fff;}
/* disabled elfinder */
.elfinder-disabled .elfinder-cwd-view-icons .elfinder-cwd-file .elfinder-cwd-filename.ui-state-hover,
.elfinder-disabled .elfinder-cwd table td.ui-state-hover { background:#dadada;}
/* statusbar */
.elfinder .elfinder-statusbar { color:#555; }
.elfinder .elfinder-statusbar a { text-decoration:none; color:#555;}
.std42-dialog .elfinder-help, .std42-dialog .elfinder-help .ui-widget-content { background:#fff;}
/* contextmenu */
.elfinder-contextmenu .ui-state-hover { background: #3875d7; color:#fff; }
.elfinder-contextmenu .ui-state-hover .elfinder-contextmenu-arrow { background-image:url('../img/arrows-active.png'); }
webmagic-avalon/webmagic-admin/src/main/webapp/static/css/jquery.cleditor.css deleted 100755 → 0
View file @ 1b886d48
.cleditorMain {border:1px solid #999; padding:0 1px 1px; background-color:white}
.cleditorMain iframe {border:none; margin:0; padding:0}
.cleditorMain textarea {border:none; margin:0; padding:0; overflow-y:scroll; font:10pt Arial,Verdana; resize:none; outline:none /* webkit grip focus */}
.cleditorToolbar {background: url('../img/toolbar.gif') repeat}
.cleditorGroup {float:left; height:26px}
.cleditorButton {float:left; width:24px; height:24px; margin:1px 0 1px 0; background: url('../img/buttons.gif')}
.cleditorDisabled {opacity:0.3; filter:alpha(opacity=30)}
.cleditorDivider {float:left; width:1px; height:23px; margin:1px 0 1px 0; background:#CCC}
.cleditorPopup {border:solid 1px #999; background-color:white; position:absolute; font:10pt Arial,Verdana; cursor:default; z-index:10000}
.cleditorList div {padding:2px 4px 2px 4px}
.cleditorList p,
.cleditorList h1,
.cleditorList h2,
.cleditorList h3,
.cleditorList h4,
.cleditorList h5,
.cleditorList h6,
.cleditorList font {padding:0; margin:0; background-color:Transparent}
.cleditorColor {width:150px; padding:1px 0 0 1px}
.cleditorColor div {float:left; width:14px; height:14px; margin:0 1px 1px 0}
.cleditorPrompt {background-color:#F6F7F9; padding:4px; font-size:8.5pt}
.cleditorPrompt input,
.cleditorPrompt textarea {font:8.5pt Arial,Verdana;}
.cleditorMsg {background-color:#FDFCEE; width:150px; padding:4px; font-size:8.5pt}
webmagic-avalon/webmagic-avalon-common/pom.xml deleted 100644 → 0
View file @ 1b886d48
This diff is collapsed.
webmagic-avalon/webmagic-worker/README.md deleted 100644 → 0
View file @ 1b886d48
This diff is collapsed.
webmagic-avalon/webmagic-worker/pom.xml deleted 100644 → 0
View file @ 1b886d48
This diff is collapsed.
webmagic-core/pom.xml
View file @ 6d78d51f
This diff is collapsed.
webmagic-core/pom.xml.releaseBackup deleted 100644 → 0
View file @ 1b886d48
This diff is collapsed.
webmagic-extension/pom.xml
View file @ 6d78d51f
This diff is collapsed.
webmagic-extension/pom.xml.releaseBackup deleted 100644 → 0
View file @ 1b886d48
This diff is collapsed.
webmagic-samples/pom.xml
View file @ 6d78d51f
This diff is collapsed.
webmagic-saxon/pom.xml
View file @ 6d78d51f
This diff is collapsed.
webmagic-scripts/pom.xml
View file @ 6d78d51f
This diff is collapsed.
webmagic-selenium/pom.xml
View file @ 6d78d51f
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment