c

rapyd yes scrapy Deployment of ,  It's an official crawler management tool ,

It is very convenient to upload and control the operation of the crawler through it ,

install : pip install scapyd

He provided a json ,web, server

Enter... On the command line scrapyd enter ,

Give me a local request :  curl http://localhost:6800

If It's interface free linux , It can be forwarded through a port

python The path to the installation Library

vim Go to the file The default is 127.0.0.1

So you can visit ,

And install a client , pin install scrapy client

Configure service information in this file

These are the only two that need to be configured , One is the server ip, One is the name of the project ,

List available servers

Deployment project : scrapyd-deploy <target0> -p <project>--version<version>

Write push in the most efficient way :scrapyd-deploy -p Project name

Call crawler : curl http://localhost:6800/schedule.json -d project xxx -d spider=xx

xxx Project name ,

xx It's the name of the reptile , In the project spider In the catalog xx.py Name of file

Cancel the reptile : curl http://localhost:6800/cancer.json -d project xxx -d job=0000000

among 0000000 It's a reptile jobid

List all job:      curl http://localhost:6800/listjobs.json?project=xxx

among xxx Project name

List reptiles :      curl http://localhost:6800/listjobs.json?spider=xxx

Delete the project : curl http://localhost:6800/delproject.json -d projec=xxxx -d version=

check scrapy process id

Tanzhou classroom 25 class :Ph201805201 Reptile advanced The ninth lesson scrapyd Deploy ( Class notes ) More articles about

  1. Tanzhou classroom 25 class :Ph201805201 Reptile advanced The second lesson sclapy frame ( Class notes )

    win Lower installation sclapy Install first  pip install wheel py Library download address :https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted Here ...

  2. Tanzhou classroom 25 class :Ph201805201 Reptile advanced The first lesson pyspider frame ( Class notes )

    utilize wheel install S1: pip install wheelS2: Get into www.lfd.uci.edu/~gohlke/pythonlibs/,Ctrl + F lookup pycurl The package name is pycur ...

  3. Tanzhou classroom 25 class :Ph201805201 Reptile base The ninth lesson The image processing - PIL ( Class notes )

    Python The image processing -Pillow brief introduction Python Traditional image processing library PIL(Python Imaging Library ), Basically Python Standard library for processing images , Powerful , Easy to use . But by ...

  4. Tanzhou College -JavaVIP Of Javascript The advanced level of -KeKe teacher

    Tanzhou College -JavaVIP Of Javascript The advanced level of -KeKe teacher   Well done , You can learn   Here's a screenshot of the tutorial catalog : Download address :http://www.fu83.cn/thread-283-1-1.htm ...

  5. Tanzhou classroom 25 class :Ph201805201 WEB And Page writing The fourth lesson Log in to register ( Class notes )

    index.html home page <!DOCTYPE html> <html lang="en"> <head> <meta charset=& ...

  6. Tanzhou classroom 25 class :Ph201805201 WEB And Page writing The third class ( Class notes )

    index.html <!DOCTYPE html> <html lang="en"> <head> <meta charset=&quo ...

  7. Tanzhou classroom 25 class :Ph201805201 WEB And Page writing The second lesson ( Class notes )

    index.html <!DOCTYPE html> <html lang="en"> <head> <meta charset=&quo ...

  8. Tanzhou classroom 25 class :Ph201805201 WEB And Page writing The first lesson ( Class notes )

    index.html <!DOCTYPE html> <html lang="en"> <head> <meta charset=&quo ...

  9. Tanzhou classroom 25 class :Ph201805201 The tenth lesson The definition of a class , Properties and methods ( Class notes )

    The definition of a class Common properties , features , Methodist , It can be divided into one kind , And in the name of class Abc: # class Defining classes , Followed by the class name ( The rules title case ) cls_name = ' The name of this class is Abc' # In the class ...

Random recommendation

  1. Start learning red hat RHCE The classroom has 2 It's a second class , Write down your learning experience

    I finally applied for the blog of blog park . Red hat class has begun 2 Time , The records here may be out of order , Record your own learning content every day . It's convenient for you to inquire later . I've learned the following : 1. Accessing the command line Use the desktop access commands GNOME 3 ...

  2. Visual Studio CLR Profiler

    http://blogs.msdn.com/b/dotnet/archive/2013/04/04/net-memory-allocation-profiling-with-visual-studio ...

  3. solve Visual Studio 2010/2012 When debugging lock Method of file

    debugging 3dsmax plug-in unit , There is a way to avoid restarting every time you modify the plug-in code 3dsmax Methods , That is to write the exported core code in a separate DLL in , Then when the plug-in code needs to be exported LoadLibrary This DLL, After exporting FreeL ...

  4. Qt The transparent prompt box

    sketch Those who use Penguin frequently must be familiar with the prompt box of login failure , It mainly involves making the window transparent and adding icons . Prompt information . Turn off the display of buttons, etc . We can use QWidget Create a prompt box , Then set the effect we want through the style . sketch effect ...

  5. Python in for\while Usage of

    Code example board = [] for i in range(5): board.append(i) print board board = [] i = 0 while i < 5: boar ...

  6. Qt in OpenGL Preliminary use of

    Results Preview : One . Code 5 File //glwidget.h #ifndef GLWIDGET_H #define GLWIDGET_H #include <QGLWidget> class G ...

  7. 【ArcGIS 10.2 New characteristics 】ArcGIS 10.2 for Server common problem

    1.ArcGIS 10.2 Is there a beta program ? 10.2 There is no formal beta program , But we plan to release two early cuts Software to international distributors . Selected users and partners . 2. Microsoft Azure The cloud ArcGIS forServe ...

  8. The first 16 Chapter Observer mode (Oberver Pattern)

    original text    The first 16 Chapter Observer mode (Oberver Pattern) Observer mode   summary :   In the process of software construction , We need to create a kind of “ Notification dependency ” —— An object ( Target audience ) The state of , All of them depend on ...

  9. WDA Configuration of

    WDA Configuration of SAP Our technology is always very complicated ,WDA The configuration involved is as follows : 1. Internet Communication Manager confirm ICM Provided in HTTP/HTTPS Running normally . Tcode: ...

  10. eclipse To solve git Branch merge conflict

    Conflict scenarios : stay master There are files on the branch student.py. stay master Add a new one dev Branch stay dev Modify the file on the branch student.py. Add function def d():, and commit; stay master ...